[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:18:01 dist-test-slave-zwbb krb5kdc[31800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:18:01 dist-test-slave-zwbb krb5kdc[31800](info): set up 2 sockets
Sep 24 15:18:01 dist-test-slave-zwbb krb5kdc[31800](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 15:18:02.792186 31793 mini_kdc.cc:121] Time spent starting KDC: real 2.463s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:18:02 dist-test-slave-zwbb krb5kdc[31800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:18:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:18:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:18:02.906548 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:41561
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:41561
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:02.991833 31813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:02.992038 31813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:02.992059 31813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:02.993191 31813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:18:02.993232 31813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:02.993247 31813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:02.993259 31813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:18:02.993271 31813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:18:02.994779 31813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:41561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:41561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:02.995019 31813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:02.995236 31813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:03.004403 31818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:03.004478 31821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:03.004464 31819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:03.004482 31813 server_base.cc:1034] running on GCE node
I20260924 15:18:03.004801 31813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:03.005136 31813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:03.006291 31813 hybrid_clock.cc:648] HybridClock initialized: now 1790263083006277 us; error 28 us; skew 500 ppm
Sep 24 15:18:03 dist-test-slave-zwbb krb5kdc[31800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263083, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:03.008781 31813 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:18:03.009179 31813 webserver.cc:460] Webserver started at http://127.31.12.126:40391/ using document root <none> and password file <none>
I20260924 15:18:03.009356 31813 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:03.009397 31813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:03.009533 31813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:03.010671 31813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "bd0aedb4d13d476d99f00d69ba8a0e8a"
format_stamp: "Formatted at 2026-09-24 15:18:03 on dist-test-slave-zwbb"
server_key: "b3b1f8daed15724acf93878da6c81be8"
server_key_iv: "a60bf13fdcf26d0273dbf761bfdb25c3"
server_key_version: "encryptionkey@0"
I20260924 15:18:03.012146 31813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "bd0aedb4d13d476d99f00d69ba8a0e8a"
format_stamp: "Formatted at 2026-09-24 15:18:03 on dist-test-slave-zwbb"
server_key: "b3b1f8daed15724acf93878da6c81be8"
server_key_iv: "a60bf13fdcf26d0273dbf761bfdb25c3"
server_key_version: "encryptionkey@0"
I20260924 15:18:03.013417 31813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:03.014197 31828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:03.014384 31813 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:03.014489 31813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "bd0aedb4d13d476d99f00d69ba8a0e8a"
format_stamp: "Formatted at 2026-09-24 15:18:03 on dist-test-slave-zwbb"
server_key: "b3b1f8daed15724acf93878da6c81be8"
server_key_iv: "a60bf13fdcf26d0273dbf761bfdb25c3"
server_key_version: "encryptionkey@0"
I20260924 15:18:03.014561 31813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:03.030337 31813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:03.032600 31813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:03.032753 31813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:03.036867 31813 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:41561
I20260924 15:18:03.036906 31881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:41561 every 8 connection(s)
I20260924 15:18:03.037287 31813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:18:03.038259 31882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:03.040551 31882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a: Bootstrap starting.
I20260924 15:18:03.041155 31882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:03.041162 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 31813
I20260924 15:18:03.041277 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:18:03.041493 31882 log.cc:824] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:03.042094 31793 external_mini_cluster.cc:1428] Setting key 999bd2f0c73f5860e5b9ada78ce231c2
I20260924 15:18:03.042264 31882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a: No bootstrap required, opened a new log
I20260924 15:18:03.044629 31882 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0aedb4d13d476d99f00d69ba8a0e8a" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 41561 } }
I20260924 15:18:03.044813 31882 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:03.044869 31882 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd0aedb4d13d476d99f00d69ba8a0e8a, State: Initialized, Role: FOLLOWER
I20260924 15:18:03.045017 31882 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0aedb4d13d476d99f00d69ba8a0e8a" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 41561 } }
I20260924 15:18:03.045102 31882 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:18:03.045145 31882 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:18:03.045193 31882 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:03.046329 31882 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0aedb4d13d476d99f00d69ba8a0e8a" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 41561 } }
I20260924 15:18:03.046458 31882 leader_election.cc:302] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd0aedb4d13d476d99f00d69ba8a0e8a; no voters:
I20260924 15:18:03.046633 31882 leader_election.cc:288] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:18:03.046685 31887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:03.046870 31887 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a [term 1 LEADER]: Becoming Leader. State: Replica: bd0aedb4d13d476d99f00d69ba8a0e8a, State: Running, Role: LEADER
I20260924 15:18:03.046877 31882 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:18:03.046998 31887 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0aedb4d13d476d99f00d69ba8a0e8a" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 41561 } }
I20260924 15:18:03.047376 31887 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd0aedb4d13d476d99f00d69ba8a0e8a. Latest consensus state: current_term: 1 leader_uuid: "bd0aedb4d13d476d99f00d69ba8a0e8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0aedb4d13d476d99f00d69ba8a0e8a" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 41561 } } }
I20260924 15:18:03.047385 31888 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd0aedb4d13d476d99f00d69ba8a0e8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd0aedb4d13d476d99f00d69ba8a0e8a" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 41561 } } }
I20260924 15:18:03.047539 31887 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:03.047546 31888 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:03.047791 31896 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:18:03.048614 31896 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:18:03.050493 31896 catalog_manager.cc:1347] Generated new cluster ID: 622e9c0ae02243ceac79167dcba34696
I20260924 15:18:03.050547 31896 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:18:03 dist-test-slave-zwbb krb5kdc[31800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:03.058321 31885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:03.042985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52988 (local address 127.31.12.126:41561)
0924 15:18:03.043153 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:03.043162 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:03.043506 (+ 344us) server_negotiation.cc:407] Connection header received
0924 15:18:03.046002 (+ 2496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:03.046008 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:03.046050 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:03.046160 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:03.048482 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:03.049884 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:03.051111 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:03.051589 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:03.051712 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:03.054976 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:03.054979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:03.054979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:03.054988 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:03.056462 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:03.056864 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:03.056866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:03.056867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:03.056970 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:03.057208 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:03.057209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:03.057209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:03.057414 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:03.057606 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:03.057858 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 15:18:03.058008 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260924 15:18:03.084734 31896 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:18:03.085307 31896 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:18:03.088043 31896 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd0aedb4d13d476d99f00d69ba8a0e8a: Generated new TSK 0
I20260924 15:18:03.088227 31896 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
I20260924 15:18:03.120111 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:41561
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:03.214902 31909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:03.215099 31909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:03.215122 31909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:03.216164 31909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:03.216197 31909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:03.216223 31909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:18:03.217649 31909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:41561
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:03.217824 31909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:03.217998 31909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:03.222921 31914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:03.222995 31917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:03.222934 31915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:03.223059 31909 server_base.cc:1034] running on GCE node
I20260924 15:18:03.223278 31909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:03.223543 31909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:03.224717 31909 hybrid_clock.cc:648] HybridClock initialized: now 1790263083224695 us; error 29 us; skew 500 ppm
Sep 24 15:18:03 dist-test-slave-zwbb krb5kdc[31800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263083, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:03.227310 31909 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:18:03.227689 31909 webserver.cc:460] Webserver started at http://127.31.12.65:46271/ using document root <none> and password file <none>
I20260924 15:18:03.227866 31909 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:03.227906 31909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:03.227996 31909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:03.228994 31909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "ed8729f25668425fad22a7eeda5638ea"
format_stamp: "Formatted at 2026-09-24 15:18:03 on dist-test-slave-zwbb"
server_key: "f77335b9a89f20901a67ee7a469e17ca"
server_key_iv: "49ab1cc270aba77126b3e774cbb93f29"
server_key_version: "encryptionkey@0"
I20260924 15:18:03.229300 31909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "ed8729f25668425fad22a7eeda5638ea"
format_stamp: "Formatted at 2026-09-24 15:18:03 on dist-test-slave-zwbb"
server_key: "f77335b9a89f20901a67ee7a469e17ca"
server_key_iv: "49ab1cc270aba77126b3e774cbb93f29"
server_key_version: "encryptionkey@0"
I20260924 15:18:03.230623 31909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:03.231285 31924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:03.231438 31909 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:03.231532 31909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "ed8729f25668425fad22a7eeda5638ea"
format_stamp: "Formatted at 2026-09-24 15:18:03 on dist-test-slave-zwbb"
server_key: "f77335b9a89f20901a67ee7a469e17ca"
server_key_iv: "49ab1cc270aba77126b3e774cbb93f29"
server_key_version: "encryptionkey@0"
I20260924 15:18:03.231588 31909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:03.244553 31909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:03.246472 31909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:03.246588 31909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:03.246763 31909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:03.247110 31909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:03.247143 31909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:03.247195 31909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:03.247222 31909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:03.252986 31909 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:37713
I20260924 15:18:03.253031 32039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:37713 every 8 connection(s)
I20260924 15:18:03.253374 31909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:18:03.255187 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 31909
I20260924 15:18:03.255259 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:18:03.255412 31793 external_mini_cluster.cc:1428] Setting key dd591f9382b50aba304dc4506cb43de0
Sep 24 15:18:03 dist-test-slave-zwbb krb5kdc[31800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263083, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:03.263900 31885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:03.254144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:54695 (local address 127.31.12.126:41561)
0924 15:18:03.254209 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:03.254211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:03.254403 (+ 192us) server_negotiation.cc:407] Connection header received
0924 15:18:03.254944 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:03.254945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:03.254959 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:03.255012 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:03.256192 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:03.256701 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:03.258116 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:03.258316 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:03.258379 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:03.260714 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:03.260716 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:03.260716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:03.260724 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:03.262412 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:03.262841 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:03.262843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:03.262843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:03.262919 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:03.263158 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:03.263160 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:03.263160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:03.263319 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:03.263397 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:03.263689 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 15:18:03.263782 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:18:03.263918 32042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:03.254127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:41561 (local address 127.31.12.65:54695)
0924 15:18:03.254283 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:03.254319 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:03.254832 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:03.255090 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:03.255094 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:03.255835 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:03.256093 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:03.256099 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:03.256817 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:03.256818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:03.258026 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:03.258029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:03.258415 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:03.258417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:03.258529 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:03.258991 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:03.259001 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:03.260618 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:03.262518 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:03.262525 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:03.262527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:03.262757 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:03.262991 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:03.262991 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:03.262992 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:03.263089 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:03.263414 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:03.263421 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:03.263542 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 15:18:03.263637 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 15:18:03.263759 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260924 15:18:03.264493 32040 heartbeater.cc:347] Connected to a master server at 127.31.12.126:41561
I20260924 15:18:03.264598 32040 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.31.12.66@KRBTEST.COM; defaulting to no policy
I20260924 15:18:03.264786 32040 heartbeater.cc:511] Master 127.31.12.126:41561 requested a full tablet report, sending...
I20260924 15:18:03.265319 31845 ts_manager.cc:194] Registered new tserver with Master: ed8729f25668425fad22a7eeda5638ea (127.31.12.65:37713)
I20260924 15:18:03.266068 31845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:54695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
I20260924 15:18:03.281781 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:41561
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:03.362282 32045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:03.362458 32045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:03.362481 32045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:03.363561 32045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:03.363593 32045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:03.363618 32045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:18:03.364964 32045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:41561
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:03.365136 32045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:03.365299 32045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:03.370139 32053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:03.370157 32050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:03.370186 32045 server_base.cc:1034] running on GCE node
W20260924 15:18:03.370154 32051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:03.370565 32045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:03.370807 32045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:03.371976 32045 hybrid_clock.cc:648] HybridClock initialized: now 1790263083371975 us; error 37 us; skew 500 ppm
Sep 24 15:18:03 dist-test-slave-zwbb krb5kdc[31800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263083, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:03.374367 32045 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:18:03.374825 32045 webserver.cc:460] Webserver started at http://127.31.12.66:46099/ using document root <none> and password file <none>
I20260924 15:18:03.375065 32045 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:03.375133 32045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:03.375267 32045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:03.376279 32045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "474ee8142d894845b28cf1c825addf01"
format_stamp: "Formatted at 2026-09-24 15:18:03 on dist-test-slave-zwbb"
server_key: "c0650f5d1404b3104a2b4fe6eacd5a09"
server_key_iv: "af096b239687b06f905c572100a16345"
server_key_version: "encryptionkey@0"
I20260924 15:18:03.376641 32045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "474ee8142d894845b28cf1c825addf01"
format_stamp: "Formatted at 2026-09-24 15:18:03 on dist-test-slave-zwbb"
server_key: "c0650f5d1404b3104a2b4fe6eacd5a09"
server_key_iv: "af096b239687b06f905c572100a16345"
server_key_version: "encryptionkey@0"
I20260924 15:18:03.378078 32045 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:18:03.378856 32060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:03.379056 32045 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:03.379145 32045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "474ee8142d894845b28cf1c825addf01"
format_stamp: "Formatted at 2026-09-24 15:18:03 on dist-test-slave-zwbb"
server_key: "c0650f5d1404b3104a2b4fe6eacd5a09"
server_key_iv: "af096b239687b06f905c572100a16345"
server_key_version: "encryptionkey@0"
I20260924 15:18:03.379240 32045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:03.395879 32045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:03.398100 32045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:03.398260 32045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:03.398495 32045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:03.398792 32045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:03.398845 32045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:03.398887 32045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:03.398916 32045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:03.405128 32045 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:34289
I20260924 15:18:03.405166 32176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:34289 every 8 connection(s)
I20260924 15:18:03.405548 32045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:18:03.406332 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 32045
I20260924 15:18:03.406426 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:18:03.406594 31793 external_mini_cluster.cc:1428] Setting key ea4f25773e2e993a600165ccc0e77023
Sep 24 15:18:03 dist-test-slave-zwbb krb5kdc[31800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263083, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.12.67@KRBTEST.COM; defaulting to no policy
I20260924 15:18:03.415385 31885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:03.406256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:59225 (local address 127.31.12.126:41561)
0924 15:18:03.406298 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:03.406300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:03.406583 (+ 283us) server_negotiation.cc:407] Connection header received
0924 15:18:03.407199 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:03.407200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:03.407211 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:03.407256 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:03.408171 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:03.408926 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:03.409960 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:03.410145 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:03.410232 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:03.412570 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:03.412571 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:03.412572 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:03.412579 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:03.414003 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:03.414418 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:03.414419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:03.414420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:03.414490 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:03.414736 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:03.414737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:03.414737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:03.414864 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:03.414942 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:03.415219 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 15:18:03.415297 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:18:03.415596 32179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:03.406254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:41561 (local address 127.31.12.66:59225)
0924 15:18:03.406441 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:03.406488 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:03.407108 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:03.407324 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:03.407327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:03.407880 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:03.408074 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:03.408079 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:03.409022 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:03.409023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:03.409883 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:03.409885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:03.410238 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:03.410239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:03.410380 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:03.410890 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:03.410898 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:03.412459 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:03.414104 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:03.414111 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:03.414113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:03.414327 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:03.414557 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:03.414558 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:03.414558 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:03.414666 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:03.414931 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:03.414938 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:03.415081 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 15:18:03.415184 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 15:18:03.415335 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":95,"threads_started":1}
I20260924 15:18:03.416087 32177 heartbeater.cc:347] Connected to a master server at 127.31.12.126:41561
I20260924 15:18:03.416164 32177 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:03.416400 32177 heartbeater.cc:511] Master 127.31.12.126:41561 requested a full tablet report, sending...
I20260924 15:18:03.416882 31845 ts_manager.cc:194] Registered new tserver with Master: 474ee8142d894845b28cf1c825addf01 (127.31.12.66:34289)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.67@KRBTEST.COM" created.
I20260924 15:18:03.417366 31845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:59225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
I20260924 15:18:03.432016 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:41561
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:03.512993 32182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:03.513190 32182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:03.513216 32182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:03.514312 32182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:03.514349 32182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:03.514382 32182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:18:03.515744 32182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:41561
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:03.515937 32182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:03.516170 32182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:03.521153 32190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:03.521168 32188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:03.521184 32182 server_base.cc:1034] running on GCE node
W20260924 15:18:03.521173 32187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:03.521682 32182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:03.521938 32182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:03.523135 32182 hybrid_clock.cc:648] HybridClock initialized: now 1790263083523122 us; error 30 us; skew 500 ppm
Sep 24 15:18:03 dist-test-slave-zwbb krb5kdc[31800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263083, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:03.525700 32182 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:18:03.526130 32182 webserver.cc:460] Webserver started at http://127.31.12.67:36635/ using document root <none> and password file <none>
I20260924 15:18:03.526355 32182 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:03.526429 32182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:03.526582 32182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:03.527644 32182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "e5f6567f0e1d4fbfa2265372ae1988fa"
format_stamp: "Formatted at 2026-09-24 15:18:03 on dist-test-slave-zwbb"
server_key: "fec356d30d6f79c3024148533c349459"
server_key_iv: "cc4821e4baa278f1f58bdd3891013176"
server_key_version: "encryptionkey@0"
I20260924 15:18:03.528069 32182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "e5f6567f0e1d4fbfa2265372ae1988fa"
format_stamp: "Formatted at 2026-09-24 15:18:03 on dist-test-slave-zwbb"
server_key: "fec356d30d6f79c3024148533c349459"
server_key_iv: "cc4821e4baa278f1f58bdd3891013176"
server_key_version: "encryptionkey@0"
I20260924 15:18:03.529532 32182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260924 15:18:03.530350 32197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:03.530529 32182 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:03.530632 32182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "e5f6567f0e1d4fbfa2265372ae1988fa"
format_stamp: "Formatted at 2026-09-24 15:18:03 on dist-test-slave-zwbb"
server_key: "fec356d30d6f79c3024148533c349459"
server_key_iv: "cc4821e4baa278f1f58bdd3891013176"
server_key_version: "encryptionkey@0"
I20260924 15:18:03.530715 32182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:03.546221 32182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:03.548265 32182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:03.548420 32182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:03.548668 32182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:03.548988 32182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:03.549041 32182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:03.549082 32182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:03.549109 32182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:03.554955 32182 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:45323
I20260924 15:18:03.554988 32312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:45323 every 8 connection(s)
I20260924 15:18:03.555301 32182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:18:03.556629 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 32182
I20260924 15:18:03.556728 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:18:03.556911 31793 external_mini_cluster.cc:1428] Setting key d4e97cf9274553e9286b6279161ebe73
Sep 24 15:18:03 dist-test-slave-zwbb krb5kdc[31800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263083, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:03.564661 31885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:03.555943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:48215 (local address 127.31.12.126:41561)
0924 15:18:03.556009 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:03.556011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:03.556245 (+ 234us) server_negotiation.cc:407] Connection header received
0924 15:18:03.556930 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:03.556932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:03.556944 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:03.557011 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:03.558314 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:03.558817 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:03.559819 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:03.560037 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:03.560120 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:03.562202 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:03.562203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:03.562203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:03.562210 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:03.563587 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:03.563908 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:03.563909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:03.563909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:03.563972 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:03.564169 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:03.564170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:03.564170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:03.564265 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:03.564314 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:03.564539 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 15:18:03.564599 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 15:18:03.564844 32315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:03.555950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:41561 (local address 127.31.12.67:48215)
0924 15:18:03.556128 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:03.556162 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:03.556803 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:03.557101 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:03.557106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:03.557914 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:03.558202 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:03.558208 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:03.558913 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:03.558914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:03.559724 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:03.559731 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:03.560131 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:03.560132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:03.560263 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:03.560675 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:03.560683 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:03.562115 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:03.563652 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:03.563656 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:03.563658 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:03.563847 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:03.564027 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:03.564028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:03.564028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:03.564117 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:03.564316 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:03.564320 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:03.564424 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 15:18:03.564513 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 15:18:03.564621 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260924 15:18:03.565367 32313 heartbeater.cc:347] Connected to a master server at 127.31.12.126:41561
I20260924 15:18:03.565539 32313 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:03.565694 32313 heartbeater.cc:511] Master 127.31.12.126:41561 requested a full tablet report, sending...
I20260924 15:18:03.566053 31845 ts_manager.cc:194] Registered new tserver with Master: e5f6567f0e1d4fbfa2265372ae1988fa (127.31.12.67:45323)
I20260924 15:18:03.566407 31845 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:48215
I20260924 15:18:03.568861 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 15:18:03 dist-test-slave-zwbb krb5kdc[31800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263083, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 15:18:03 dist-test-slave-zwbb krb5kdc[31800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263083, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.12.65@KRBTEST.COM
I20260924 15:18:03.591898 32321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:03.584180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52440 (local address 127.31.12.65:37713)
0924 15:18:03.584353 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:03.584359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:03.584369 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:18:03.584397 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:03.584399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:03.584415 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:03.584513 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:03.585113 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:03.585960 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:03.586725 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:03.587081 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:03.587163 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:03.589553 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:03.589561 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:03.589561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:03.589567 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:03.591004 (+ 1437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:03.591293 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:03.591295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:03.591295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:03.591345 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:03.591499 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:03.591499 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:03.591499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:03.591607 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:03.591717 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:03.591768 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 15:18:03.591825 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
W20260924 15:18:03.592264 31953 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52440
Sep 24 15:18:03 dist-test-slave-zwbb krb5kdc[31800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 15:18:03 dist-test-slave-zwbb krb5kdc[31800](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.65@KRBTEST.COM
I20260924 15:18:03.614207 32321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:03.606871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.31.12.65:37713)
0924 15:18:03.606954 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:03.606957 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:03.607064 (+ 107us) server_negotiation.cc:407] Connection header received
0924 15:18:03.607179 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:03.607180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:03.607193 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:03.607280 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:03.607915 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:03.608460 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:03.609294 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:03.609614 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:03.609709 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:03.612032 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:03.612033 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:03.612033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:03.612039 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:03.613210 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:03.613556 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:03.613557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:03.613557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:03.613621 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:03.613778 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:03.613779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:03.613779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:03.613853 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:03.613927 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:03.614081 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 15:18:03.614142 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 15:18:03.614765 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 31909
I20260924 15:18:03.620440 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 32045
I20260924 15:18:03.627362 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 32182
I20260924 15:18:03.633221 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 31813
2026-09-24T15:18:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3336 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:18:03 dist-test-slave-zwbb krb5kdc[32328](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:18:03 dist-test-slave-zwbb krb5kdc[32328](info): set up 2 sockets
Sep 24 15:18:03 dist-test-slave-zwbb krb5kdc[32328](info): commencing operation
krb5kdc: starting...
W20260924 15:18:04.679319 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:18:04 dist-test-slave-zwbb krb5kdc[32328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:18:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:18:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:18:04.796917 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:41933
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:33387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:41933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:04.876140 32341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:04.876368 32341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:04.876415 32341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:04.877576 32341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:18:04.877642 32341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:04.877666 32341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:04.877689 32341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:18:04.877731 32341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:18:04.879166 32341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:41933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:41933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:04.879437 32341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:04.879652 32341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:04.884631 32346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:04.884644 32347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:04.884698 32341 server_base.cc:1034] running on GCE node
W20260924 15:18:04.884639 32349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:04.885058 32341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:04.885318 32341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:04.886489 32341 hybrid_clock.cc:648] HybridClock initialized: now 1790263084886476 us; error 32 us; skew 500 ppm
Sep 24 15:18:04 dist-test-slave-zwbb krb5kdc[32328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263084, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:04.895067 32341 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:18:04.895437 32341 webserver.cc:460] Webserver started at http://127.31.12.126:33839/ using document root <none> and password file <none>
I20260924 15:18:04.895711 32341 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:04.895756 32341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:04.895919 32341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:04.896965 32341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "3f3a21b00c49486bbe8c1a719f8a9b9c"
format_stamp: "Formatted at 2026-09-24 15:18:04 on dist-test-slave-zwbb"
server_key: "1161b4b54957d8ad254f43c2da0326d3"
server_key_iv: "a4007a599a40d402a16a4823f0bf0517"
server_key_version: "encryptionkey@0"
I20260924 15:18:04.897331 32341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "3f3a21b00c49486bbe8c1a719f8a9b9c"
format_stamp: "Formatted at 2026-09-24 15:18:04 on dist-test-slave-zwbb"
server_key: "1161b4b54957d8ad254f43c2da0326d3"
server_key_iv: "a4007a599a40d402a16a4823f0bf0517"
server_key_version: "encryptionkey@0"
I20260924 15:18:04.898656 32341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:18:04.899375 32356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:04.899570 32341 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:04.899633 32341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "3f3a21b00c49486bbe8c1a719f8a9b9c"
format_stamp: "Formatted at 2026-09-24 15:18:04 on dist-test-slave-zwbb"
server_key: "1161b4b54957d8ad254f43c2da0326d3"
server_key_iv: "a4007a599a40d402a16a4823f0bf0517"
server_key_version: "encryptionkey@0"
I20260924 15:18:04.899720 32341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:04.907907 32341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:04.909657 32341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:04.909806 32341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:04.913583 32341 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:41933
I20260924 15:18:04.913632 32408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:41933 every 8 connection(s)
I20260924 15:18:04.913986 32341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:18:04.914784 32409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:04.916973 32409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c: Bootstrap starting.
I20260924 15:18:04.917629 32409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:04.917917 32409 log.cc:824] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:04.918576 32409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c: No bootstrap required, opened a new log
I20260924 15:18:04.920145 32409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3a21b00c49486bbe8c1a719f8a9b9c" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 41933 } }
I20260924 15:18:04.920279 32409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:04.920300 32409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f3a21b00c49486bbe8c1a719f8a9b9c, State: Initialized, Role: FOLLOWER
I20260924 15:18:04.920470 32409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3a21b00c49486bbe8c1a719f8a9b9c" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 41933 } }
I20260924 15:18:04.920533 32409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:18:04.920554 32409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:18:04.920622 32409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:04.921123 32409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3a21b00c49486bbe8c1a719f8a9b9c" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 41933 } }
I20260924 15:18:04.921159 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 32341
I20260924 15:18:04.921258 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:18:04.921241 32409 leader_election.cc:302] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f3a21b00c49486bbe8c1a719f8a9b9c; no voters:
I20260924 15:18:04.921434 32409 leader_election.cc:288] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:18:04.921428 31793 external_mini_cluster.cc:1428] Setting key 3b4b9e9f637df2870f6569e8f0290cf9
I20260924 15:18:04.921485 32412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:04.921639 32412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c [term 1 LEADER]: Becoming Leader. State: Replica: 3f3a21b00c49486bbe8c1a719f8a9b9c, State: Running, Role: LEADER
I20260924 15:18:04.921721 32409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:18:04.921777 32412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3a21b00c49486bbe8c1a719f8a9b9c" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 41933 } }
I20260924 15:18:04.922151 32414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f3a21b00c49486bbe8c1a719f8a9b9c. Latest consensus state: current_term: 1 leader_uuid: "3f3a21b00c49486bbe8c1a719f8a9b9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3a21b00c49486bbe8c1a719f8a9b9c" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 41933 } } }
I20260924 15:18:04.922282 32413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f3a21b00c49486bbe8c1a719f8a9b9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f3a21b00c49486bbe8c1a719f8a9b9c" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 41933 } } }
I20260924 15:18:04.922369 32413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:04.922322 32414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:04.922748 32421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:18:04.923306 32421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:18:04.924840 32421 catalog_manager.cc:1347] Generated new cluster ID: 331f9119adce4a74b5c3deca3ecba6bf
I20260924 15:18:04.924960 32421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:18:04 dist-test-slave-zwbb krb5kdc[32328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:04.934242 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:04.921978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56346 (local address 127.31.12.126:41933)
0924 15:18:04.922516 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:04.922762 (+ 246us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:04.922797 (+ 35us) server_negotiation.cc:407] Connection header received
0924 15:18:04.922913 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:04.922920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:04.922975 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:04.923977 (+ 1002us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:04.924075 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:04.925517 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:04.926544 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:04.927136 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:04.927230 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:04.929989 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:04.929993 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:04.929993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:04.930005 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:04.932590 (+ 2585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:04.932967 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:04.932971 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:04.932971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:04.933088 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:04.933291 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:04.933292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:04.933292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:04.933633 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:04.933819 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:04.933890 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 15:18:04.934022 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260924 15:18:04.938535 32421 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:18:04.939005 32421 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:18:04.948263 32421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f3a21b00c49486bbe8c1a719f8a9b9c: Generated new TSK 0
I20260924 15:18:04.948468 32421 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
I20260924 15:18:04.976894 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:41933
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:33387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:05.059769 32436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:05.059990 32436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:05.060014 32436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:05.061137 32436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:05.061180 32436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:05.061233 32436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:18:05.062948 32436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:41933
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:05.063199 32436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:05.063460 32436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:18:05.068338 32436 server_base.cc:1034] running on GCE node
W20260924 15:18:05.068305 32444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:05.068305 32442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:05.068300 32441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:05.068774 32436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:05.069067 32436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:05.070243 32436 hybrid_clock.cc:648] HybridClock initialized: now 1790263085070237 us; error 36 us; skew 500 ppm
Sep 24 15:18:05 dist-test-slave-zwbb krb5kdc[32328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263085, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:05.072734 32436 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:18:05.073164 32436 webserver.cc:460] Webserver started at http://127.31.12.65:42451/ using document root <none> and password file <none>
I20260924 15:18:05.073406 32436 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:05.073494 32436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:05.073642 32436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:05.074651 32436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "1894da50fe36470a9436c3339b564047"
format_stamp: "Formatted at 2026-09-24 15:18:05 on dist-test-slave-zwbb"
server_key: "a83a9b5566fe83b97eb7d4eeab43a863"
server_key_iv: "03b4f414d9018e69a5317126f9b00fd2"
server_key_version: "encryptionkey@0"
I20260924 15:18:05.075001 32436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "1894da50fe36470a9436c3339b564047"
format_stamp: "Formatted at 2026-09-24 15:18:05 on dist-test-slave-zwbb"
server_key: "a83a9b5566fe83b97eb7d4eeab43a863"
server_key_iv: "03b4f414d9018e69a5317126f9b00fd2"
server_key_version: "encryptionkey@0"
I20260924 15:18:05.076279 32436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:18:05.076970 32451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:05.077126 32436 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:18:05.077230 32436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "1894da50fe36470a9436c3339b564047"
format_stamp: "Formatted at 2026-09-24 15:18:05 on dist-test-slave-zwbb"
server_key: "a83a9b5566fe83b97eb7d4eeab43a863"
server_key_iv: "03b4f414d9018e69a5317126f9b00fd2"
server_key_version: "encryptionkey@0"
I20260924 15:18:05.077306 32436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:05.088935 32436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:05.090778 32436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:05.090939 32436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:05.091157 32436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:05.091459 32436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:05.091513 32436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:05.091554 32436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:05.091581 32436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:05.097226 32436 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:39495
I20260924 15:18:05.097307 32566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:39495 every 8 connection(s)
I20260924 15:18:05.097641 32436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:18:05.102272 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 32436
I20260924 15:18:05.102377 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:18:05.102592 31793 external_mini_cluster.cc:1428] Setting key 8210b17f4cd4a993549dfec481698249
Sep 24 15:18:05 dist-test-slave-zwbb krb5kdc[32328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263085, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.12.66@KRBTEST.COM; defaulting to no policy
I20260924 15:18:05.110379 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:05.098396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:57819 (local address 127.31.12.126:41933)
0924 15:18:05.098456 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:05.098458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:05.098672 (+ 214us) server_negotiation.cc:407] Connection header received
0924 15:18:05.102704 (+ 4032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:05.102706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:05.102725 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:05.102780 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:05.103513 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.104062 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.105100 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.105300 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.105389 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:05.107507 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:05.107509 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:05.107509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:05.107516 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:05.108948 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:05.109377 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:05.109378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:05.109378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:05.109472 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:05.109709 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:05.109710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:05.109710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:05.109824 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:05.109901 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:05.110183 (+ 282us) server_negotiation.cc:299] Negotiation successful
0924 15:18:05.110280 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:18:05.110543 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:05.098371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:41933 (local address 127.31.12.65:57819)
0924 15:18:05.098539 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:05.098576 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:05.102584 (+ 4008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:05.102856 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:05.102860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:05.103198 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:05.103395 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:05.103400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.104165 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.104166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:05.105005 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:05.105007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.105400 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.105402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:05.105557 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:05.106081 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:05.106090 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:05.107410 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:05.109052 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:05.109060 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:05.109062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:05.109291 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:05.109525 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:05.109526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:05.109526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:05.109640 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:05.109893 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:05.109900 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:05.110044 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 15:18:05.110149 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 15:18:05.110306 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":95,"threads_started":1}
I20260924 15:18:05.111099 32567 heartbeater.cc:347] Connected to a master server at 127.31.12.126:41933
I20260924 15:18:05.111202 32567 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:05.111388 32567 heartbeater.cc:511] Master 127.31.12.126:41933 requested a full tablet report, sending...
I20260924 15:18:05.111912 32373 ts_manager.cc:194] Registered new tserver with Master: 1894da50fe36470a9436c3339b564047 (127.31.12.65:39495)
I20260924 15:18:05.112521 32373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:57819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
I20260924 15:18:05.127863 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:41933
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:33387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:05.209540 32572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:05.209739 32572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:05.209764 32572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:05.210933 32572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:05.210975 32572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:05.211001 32572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:18:05.212484 32572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:41933
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:05.212718 32572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:05.212972 32572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:05.217819 32580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:05.217854 32577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:05.218019 32572 server_base.cc:1034] running on GCE node
W20260924 15:18:05.218142 32578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:05.218385 32572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:05.218593 32572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:05.219731 32572 hybrid_clock.cc:648] HybridClock initialized: now 1790263085219717 us; error 28 us; skew 500 ppm
Sep 24 15:18:05 dist-test-slave-zwbb krb5kdc[32328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263085, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:05.222435 32572 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:18:05.222898 32572 webserver.cc:460] Webserver started at http://127.31.12.66:36381/ using document root <none> and password file <none>
I20260924 15:18:05.223194 32572 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:05.223264 32572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:05.223410 32572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:05.224489 32572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "5321774d284e418eac210a7735be4f56"
format_stamp: "Formatted at 2026-09-24 15:18:05 on dist-test-slave-zwbb"
server_key: "c0f972c64c44d1bbee48190e85704bd9"
server_key_iv: "9760bc1f9da6128d749427319523209f"
server_key_version: "encryptionkey@0"
I20260924 15:18:05.224848 32572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "5321774d284e418eac210a7735be4f56"
format_stamp: "Formatted at 2026-09-24 15:18:05 on dist-test-slave-zwbb"
server_key: "c0f972c64c44d1bbee48190e85704bd9"
server_key_iv: "9760bc1f9da6128d749427319523209f"
server_key_version: "encryptionkey@0"
I20260924 15:18:05.226233 32572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:18:05.226922 32587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:05.227093 32572 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:05.227201 32572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "5321774d284e418eac210a7735be4f56"
format_stamp: "Formatted at 2026-09-24 15:18:05 on dist-test-slave-zwbb"
server_key: "c0f972c64c44d1bbee48190e85704bd9"
server_key_iv: "9760bc1f9da6128d749427319523209f"
server_key_version: "encryptionkey@0"
I20260924 15:18:05.227274 32572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:05.241801 32572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:05.244040 32572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:05.244202 32572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:05.244439 32572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:05.244741 32572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:05.244794 32572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:05.244828 32572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:05.244869 32572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:05.250625 32572 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:41615
I20260924 15:18:05.250676 32702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:41615 every 8 connection(s)
I20260924 15:18:05.250980 32572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:18:05.252737 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 32572
I20260924 15:18:05.252811 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:18:05.252985 31793 external_mini_cluster.cc:1428] Setting key ead358ec666efb91c4623324af5a61f3
Sep 24 15:18:05 dist-test-slave-zwbb krb5kdc[32328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263085, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.12.67@KRBTEST.COM; defaulting to no policy
I20260924 15:18:05.261665 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:05.251732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:48619 (local address 127.31.12.126:41933)
0924 15:18:05.251802 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:05.251804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:05.252006 (+ 202us) server_negotiation.cc:407] Connection header received
0924 15:18:05.252994 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:05.252996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:05.253011 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:05.253088 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:05.253853 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.254627 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.255908 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.256139 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.256219 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:05.258755 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:05.258756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:05.258757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:05.258763 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:05.260133 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:05.260604 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:05.260605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:05.260605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:05.260654 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:05.260935 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:05.260936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:05.260936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:05.261035 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:05.261098 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:05.261411 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 15:18:05.261568 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 15:18:05.261859 32705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:05.251708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:41933 (local address 127.31.12.66:48619)
0924 15:18:05.251873 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:05.251914 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:05.252865 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:05.253160 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:05.253165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:05.253492 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:05.253750 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:05.253756 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.254738 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.254739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:05.255811 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:05.255814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.256224 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.256225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:05.256359 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:05.256850 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:05.256858 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:05.258662 (+ 1804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:05.260220 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:05.260227 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:05.260229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:05.260529 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:05.260722 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:05.260723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:05.260723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:05.260861 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:05.261116 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:05.261122 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:05.261283 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 15:18:05.261395 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 15:18:05.261613 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260924 15:18:05.262368 32703 heartbeater.cc:347] Connected to a master server at 127.31.12.126:41933
I20260924 15:18:05.262470 32703 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:05.262611 32703 heartbeater.cc:511] Master 127.31.12.126:41933 requested a full tablet report, sending...
I20260924 15:18:05.263017 32373 ts_manager.cc:194] Registered new tserver with Master: 5321774d284e418eac210a7735be4f56 (127.31.12.66:41615)
I20260924 15:18:05.263459 32373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:48619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
I20260924 15:18:05.278434 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:41933
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:33387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:05.356436 32708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:05.356617 32708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:05.356638 32708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:05.357764 32708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:05.357801 32708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:05.357828 32708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:18:05.359356 32708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:41933
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:05.359583 32708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:05.359798 32708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:05.364560 32713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:05.364550 32714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:05.364550 32716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:05.364758 32708 server_base.cc:1034] running on GCE node
I20260924 15:18:05.365001 32708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:05.365244 32708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:05.366408 32708 hybrid_clock.cc:648] HybridClock initialized: now 1790263085366402 us; error 34 us; skew 500 ppm
Sep 24 15:18:05 dist-test-slave-zwbb krb5kdc[32328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263085, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:05.368744 32708 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:18:05.369156 32708 webserver.cc:460] Webserver started at http://127.31.12.67:38597/ using document root <none> and password file <none>
I20260924 15:18:05.369375 32708 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:05.369472 32708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:05.369624 32708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:05.370625 32708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "f6b44c830f794602b95125d0cfe97c8e"
format_stamp: "Formatted at 2026-09-24 15:18:05 on dist-test-slave-zwbb"
server_key: "9d353b54b03fa7ba2463f7353256c60e"
server_key_iv: "7b5d2d867b23fa05f22fb03f62efe478"
server_key_version: "encryptionkey@0"
I20260924 15:18:05.370987 32708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "f6b44c830f794602b95125d0cfe97c8e"
format_stamp: "Formatted at 2026-09-24 15:18:05 on dist-test-slave-zwbb"
server_key: "9d353b54b03fa7ba2463f7353256c60e"
server_key_iv: "7b5d2d867b23fa05f22fb03f62efe478"
server_key_version: "encryptionkey@0"
I20260924 15:18:05.372236 32708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:18:05.372956 32723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:05.373117 32708 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:05.373229 32708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "f6b44c830f794602b95125d0cfe97c8e"
format_stamp: "Formatted at 2026-09-24 15:18:05 on dist-test-slave-zwbb"
server_key: "9d353b54b03fa7ba2463f7353256c60e"
server_key_iv: "7b5d2d867b23fa05f22fb03f62efe478"
server_key_version: "encryptionkey@0"
I20260924 15:18:05.373324 32708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:05.410228 32708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:05.412205 32708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:05.412317 32708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:05.412490 32708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:05.412814 32708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:05.412870 32708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:05.412915 32708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:05.412954 32708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:05.418591 32708 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:40961
I20260924 15:18:05.418686 370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:40961 every 8 connection(s)
I20260924 15:18:05.418988 32708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:18:05.423072 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 32708
I20260924 15:18:05.423182 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:18:05.423360 31793 external_mini_cluster.cc:1428] Setting key b71f117e9a158d900e49dd1f187cec24
Sep 24 15:18:05 dist-test-slave-zwbb krb5kdc[32328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263085, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:05.428366 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:05.419712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:56787 (local address 127.31.12.126:41933)
0924 15:18:05.419768 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:05.419770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:05.419996 (+ 226us) server_negotiation.cc:407] Connection header received
0924 15:18:05.420872 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:05.420873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:05.420882 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:05.420941 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:05.421592 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.422363 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.423530 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.423813 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.423878 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:05.425948 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:05.425949 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:05.425949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:05.425955 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:05.427310 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:05.427647 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:05.427647 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:05.427648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:05.427696 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:05.427889 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:05.427889 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:05.427890 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:05.428001 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:05.428049 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:05.428251 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 15:18:05.428306 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 15:18:05.428609 373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:05.419686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:41933 (local address 127.31.12.67:56787)
0924 15:18:05.419838 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:05.419877 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:05.420789 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:05.420998 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:05.421001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:05.421239 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:05.421498 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:05.421504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.422442 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.422443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:05.423431 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:05.423434 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.423883 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.423884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:05.424019 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:05.424452 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:05.424459 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:05.425843 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:05.427375 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:05.427380 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:05.427381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:05.427585 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:05.427750 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:05.427750 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:05.427750 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:05.427837 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:05.428053 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:05.428057 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:05.428161 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 15:18:05.428239 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 15:18:05.428348 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260924 15:18:05.429051 371 heartbeater.cc:347] Connected to a master server at 127.31.12.126:41933
I20260924 15:18:05.429157 371 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:05.429324 371 heartbeater.cc:511] Master 127.31.12.126:41933 requested a full tablet report, sending...
I20260924 15:18:05.429731 32373 ts_manager.cc:194] Registered new tserver with Master: f6b44c830f794602b95125d0cfe97c8e (127.31.12.67:40961)
I20260924 15:18:05.430127 32373 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:56787
I20260924 15:18:05.435034 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:18:05.442690 32416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:05.436507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56356 (local address 127.31.12.126:41933)
0924 15:18:05.436542 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:05.436544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:05.436668 (+ 124us) server_negotiation.cc:407] Connection header received
0924 15:18:05.436744 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:05.436745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:05.436755 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:05.436826 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:05.437383 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.437902 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.438613 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.438802 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.438917 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:05.440079 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:05.440080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:05.440081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:05.440087 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:05.441531 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:05.441884 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:05.441886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:05.441886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:05.441947 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:05.442145 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:05.442146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:05.442147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:05.442292 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:05.442404 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:05.442517 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 15:18:05.442596 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:18:05.444800 32373 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:18:05.446463 32373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:18:05.452622 380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:05.448535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:40961 (local address 127.0.0.1:52846)
0924 15:18:05.448882 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:05.448894 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:05.448953 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:05.449344 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:05.449345 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:05.449346 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:05.449387 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:05.449389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.450695 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.450696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:05.451569 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:05.451572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.452334 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.452335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:05.452413 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:05.452426 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 15:18:05.452499 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 15:18:05.452552 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":41,"thread_start_us":51,"threads_started":1}
I20260924 15:18:05.452636 383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:05.448721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52846 (local address 127.31.12.67:40961)
0924 15:18:05.448874 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:05.448878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:05.449139 (+ 261us) server_negotiation.cc:407] Connection header received
0924 15:18:05.449174 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:05.449176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:05.449188 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:05.449268 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:05.449497 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.450554 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.451649 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.452249 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.452321 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:05.452369 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:05.452521 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 15:18:05.452563 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260924 15:18:05.452648 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:05.448600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50478 (local address 127.31.12.66:41615)
0924 15:18:05.448743 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:05.448748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:05.448874 (+ 126us) server_negotiation.cc:407] Connection header received
0924 15:18:05.449020 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:05.449023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:05.449038 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:05.449087 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:05.449405 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.450200 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.451549 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.452181 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.452216 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:05.452252 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:05.452537 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 15:18:05.452574 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260924 15:18:05.452901 379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:05.448502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:41615 (local address 127.0.0.1:50478)
0924 15:18:05.448766 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:05.448805 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:05.448912 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:05.449210 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:05.449213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:05.449215 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:05.449281 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:05.449287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.450296 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.450297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:05.451450 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:05.451453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.452329 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.452330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:05.452413 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:05.452427 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 15:18:05.452773 (+ 346us) client_negotiation.cc:240] Negotiation successful
0924 15:18:05.452828 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":2,"thread_start_us":64,"threads_started":1}
I20260924 15:18:05.453058 381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:05.448535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:39495 (local address 127.0.0.1:37308)
0924 15:18:05.449048 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:05.449055 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:05.449092 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:05.449261 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:05.449262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:05.449264 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:05.449331 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:05.449333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.450681 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.450682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:05.451845 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:05.451847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.452812 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.452813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:05.452888 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:05.452892 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:18:05.452939 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 15:18:05.452980 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":31,"thread_start_us":46,"threads_started":1}
I20260924 15:18:05.453299 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:05.448841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.31.12.65:39495)
0924 15:18:05.449008 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:05.449012 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:05.449090 (+ 78us) server_negotiation.cc:407] Connection header received
0924 15:18:05.449152 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:05.449154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:05.449168 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:05.449248 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:05.449419 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.450562 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.451946 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.452657 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.452725 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:05.452778 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:05.453185 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 15:18:05.453228 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260924 15:18:05.453604 304 tablet_service.cc:1467] Processing CreateTablet for tablet 6a46277b09054407a019a07090f7b2f9 (DEFAULT_TABLE table=test-table [id=9fa2f3b328ea41b08b2c805eccff589a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:05.454000 304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a46277b09054407a019a07090f7b2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:05.454002 32636 tablet_service.cc:1467] Processing CreateTablet for tablet 6a46277b09054407a019a07090f7b2f9 (DEFAULT_TABLE table=test-table [id=9fa2f3b328ea41b08b2c805eccff589a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:05.454236 32636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a46277b09054407a019a07090f7b2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:05.455653 32500 tablet_service.cc:1467] Processing CreateTablet for tablet 6a46277b09054407a019a07090f7b2f9 (DEFAULT_TABLE table=test-table [id=9fa2f3b328ea41b08b2c805eccff589a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:05.455973 32500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a46277b09054407a019a07090f7b2f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:05.456966 385 tablet_bootstrap.cc:492] T 6a46277b09054407a019a07090f7b2f9 P f6b44c830f794602b95125d0cfe97c8e: Bootstrap starting.
I20260924 15:18:05.457612 385 tablet_bootstrap.cc:654] T 6a46277b09054407a019a07090f7b2f9 P f6b44c830f794602b95125d0cfe97c8e: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:05.457661 386 tablet_bootstrap.cc:492] T 6a46277b09054407a019a07090f7b2f9 P 5321774d284e418eac210a7735be4f56: Bootstrap starting.
I20260924 15:18:05.458317 385 log.cc:824] T 6a46277b09054407a019a07090f7b2f9 P f6b44c830f794602b95125d0cfe97c8e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:05.458372 386 tablet_bootstrap.cc:654] T 6a46277b09054407a019a07090f7b2f9 P 5321774d284e418eac210a7735be4f56: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:05.458946 386 log.cc:824] T 6a46277b09054407a019a07090f7b2f9 P 5321774d284e418eac210a7735be4f56: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:05.459163 385 tablet_bootstrap.cc:492] T 6a46277b09054407a019a07090f7b2f9 P f6b44c830f794602b95125d0cfe97c8e: No bootstrap required, opened a new log
I20260924 15:18:05.459232 385 ts_tablet_manager.cc:1397] T 6a46277b09054407a019a07090f7b2f9 P f6b44c830f794602b95125d0cfe97c8e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:18:05.459807 386 tablet_bootstrap.cc:492] T 6a46277b09054407a019a07090f7b2f9 P 5321774d284e418eac210a7735be4f56: No bootstrap required, opened a new log
I20260924 15:18:05.459899 386 ts_tablet_manager.cc:1397] T 6a46277b09054407a019a07090f7b2f9 P 5321774d284e418eac210a7735be4f56: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:18:05.461143 385 raft_consensus.cc:348] T 6a46277b09054407a019a07090f7b2f9 P f6b44c830f794602b95125d0cfe97c8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5321774d284e418eac210a7735be4f56" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 41615 } } peers { permanent_uuid: "1894da50fe36470a9436c3339b564047" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 39495 } } peers { permanent_uuid: "f6b44c830f794602b95125d0cfe97c8e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 40961 } }
I20260924 15:18:05.461320 385 raft_consensus.cc:374] T 6a46277b09054407a019a07090f7b2f9 P f6b44c830f794602b95125d0cfe97c8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:05.461355 385 raft_consensus.cc:729] T 6a46277b09054407a019a07090f7b2f9 P f6b44c830f794602b95125d0cfe97c8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6b44c830f794602b95125d0cfe97c8e, State: Initialized, Role: FOLLOWER
I20260924 15:18:05.461494 385 consensus_queue.cc:260] T 6a46277b09054407a019a07090f7b2f9 P f6b44c830f794602b95125d0cfe97c8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5321774d284e418eac210a7735be4f56" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 41615 } } peers { permanent_uuid: "1894da50fe36470a9436c3339b564047" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 39495 } } peers { permanent_uuid: "f6b44c830f794602b95125d0cfe97c8e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 40961 } }
I20260924 15:18:05.461787 385 ts_tablet_manager.cc:1428] T 6a46277b09054407a019a07090f7b2f9 P f6b44c830f794602b95125d0cfe97c8e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 15:18:05.461805 371 heartbeater.cc:503] Master 127.31.12.126:41933 was elected leader, sending a full tablet report...
I20260924 15:18:05.461984 388 tablet_bootstrap.cc:492] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047: Bootstrap starting.
I20260924 15:18:05.461884 386 raft_consensus.cc:348] T 6a46277b09054407a019a07090f7b2f9 P 5321774d284e418eac210a7735be4f56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5321774d284e418eac210a7735be4f56" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 41615 } } peers { permanent_uuid: "1894da50fe36470a9436c3339b564047" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 39495 } } peers { permanent_uuid: "f6b44c830f794602b95125d0cfe97c8e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 40961 } }
I20260924 15:18:05.462049 386 raft_consensus.cc:374] T 6a46277b09054407a019a07090f7b2f9 P 5321774d284e418eac210a7735be4f56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:05.462131 386 raft_consensus.cc:729] T 6a46277b09054407a019a07090f7b2f9 P 5321774d284e418eac210a7735be4f56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5321774d284e418eac210a7735be4f56, State: Initialized, Role: FOLLOWER
I20260924 15:18:05.462250 386 consensus_queue.cc:260] T 6a46277b09054407a019a07090f7b2f9 P 5321774d284e418eac210a7735be4f56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5321774d284e418eac210a7735be4f56" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 41615 } } peers { permanent_uuid: "1894da50fe36470a9436c3339b564047" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 39495 } } peers { permanent_uuid: "f6b44c830f794602b95125d0cfe97c8e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 40961 } }
I20260924 15:18:05.462575 386 ts_tablet_manager.cc:1428] T 6a46277b09054407a019a07090f7b2f9 P 5321774d284e418eac210a7735be4f56: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:18:05.462638 32703 heartbeater.cc:503] Master 127.31.12.126:41933 was elected leader, sending a full tablet report...
I20260924 15:18:05.462659 388 tablet_bootstrap.cc:654] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:05.463275 388 log.cc:824] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:05.463958 388 tablet_bootstrap.cc:492] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047: No bootstrap required, opened a new log
I20260924 15:18:05.464041 388 ts_tablet_manager.cc:1397] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:18:05.465541 388 raft_consensus.cc:348] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5321774d284e418eac210a7735be4f56" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 41615 } } peers { permanent_uuid: "1894da50fe36470a9436c3339b564047" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 39495 } } peers { permanent_uuid: "f6b44c830f794602b95125d0cfe97c8e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 40961 } }
I20260924 15:18:05.465694 388 raft_consensus.cc:374] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:05.465740 388 raft_consensus.cc:729] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1894da50fe36470a9436c3339b564047, State: Initialized, Role: FOLLOWER
I20260924 15:18:05.465871 388 consensus_queue.cc:260] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5321774d284e418eac210a7735be4f56" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 41615 } } peers { permanent_uuid: "1894da50fe36470a9436c3339b564047" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 39495 } } peers { permanent_uuid: "f6b44c830f794602b95125d0cfe97c8e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 40961 } }
I20260924 15:18:05.466121 388 ts_tablet_manager.cc:1428] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:18:05.466150 32567 heartbeater.cc:503] Master 127.31.12.126:41933 was elected leader, sending a full tablet report...
W20260924 15:18:05.501619 32704 tablet.cc:2369] T 6a46277b09054407a019a07090f7b2f9 P 5321774d284e418eac210a7735be4f56: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:18:05.598383 32568 tablet.cc:2369] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:18:05.669612 372 tablet.cc:2369] T 6a46277b09054407a019a07090f7b2f9 P f6b44c830f794602b95125d0cfe97c8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:18:05.743727 393 raft_consensus.cc:482] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:18:05.743877 393 raft_consensus.cc:504] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5321774d284e418eac210a7735be4f56" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 41615 } } peers { permanent_uuid: "1894da50fe36470a9436c3339b564047" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 39495 } } peers { permanent_uuid: "f6b44c830f794602b95125d0cfe97c8e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 40961 } }
I20260924 15:18:05.744249 393 leader_election.cc:288] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5321774d284e418eac210a7735be4f56 (127.31.12.66:41615), f6b44c830f794602b95125d0cfe97c8e (127.31.12.67:40961)
I20260924 15:18:05.748342 383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:05.744534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:54071 (local address 127.31.12.67:40961)
0924 15:18:05.744583 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:05.744585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:05.744626 (+ 41us) server_negotiation.cc:407] Connection header received
0924 15:18:05.744727 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:05.744728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:05.744737 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:05.744794 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:05.745022 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.745787 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.747413 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.747979 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.748059 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:05.748067 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:05.748240 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 15:18:05.748276 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 15:18:05.748342 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:05.744565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:38437 (local address 127.31.12.66:41615)
0924 15:18:05.744604 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:05.744606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:05.744618 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:18:05.744727 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:05.744728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:05.744739 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:05.744794 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:05.745024 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.745786 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.747413 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.747994 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.748059 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:05.748067 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:05.748240 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 15:18:05.748276 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:18:05.748397 395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:05.744446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:41615 (local address 127.31.12.65:38437)
0924 15:18:05.744562 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:05.744579 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:05.744635 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:05.744813 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:05.744813 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:05.744816 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:05.744948 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:05.744950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.745910 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.745911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:05.747313 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:05.747316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.748080 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.748081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:05.748159 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:05.748164 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:18:05.748222 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 15:18:05.748276 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":116,"thread_start_us":54,"threads_started":1}
I20260924 15:18:05.748400 394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:05.744434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:40961 (local address 127.31.12.65:54071)
0924 15:18:05.744562 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:05.744579 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:05.744635 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:05.744812 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:05.744813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:05.744816 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:05.744936 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:05.744939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.745909 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.745911 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:05.747313 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:05.747316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.748078 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.748079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:05.748159 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:05.748165 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 15:18:05.748222 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:18:05.748276 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":109,"thread_start_us":63,"threads_started":1}
I20260924 15:18:05.748718 32657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a46277b09054407a019a07090f7b2f9" candidate_uuid: "1894da50fe36470a9436c3339b564047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5321774d284e418eac210a7735be4f56" is_pre_election: true
I20260924 15:18:05.748903 32657 raft_consensus.cc:2454] T 6a46277b09054407a019a07090f7b2f9 P 5321774d284e418eac210a7735be4f56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1894da50fe36470a9436c3339b564047 in term 0.
I20260924 15:18:05.748883 325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a46277b09054407a019a07090f7b2f9" candidate_uuid: "1894da50fe36470a9436c3339b564047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b44c830f794602b95125d0cfe97c8e" is_pre_election: true
I20260924 15:18:05.749024 325 raft_consensus.cc:2454] T 6a46277b09054407a019a07090f7b2f9 P f6b44c830f794602b95125d0cfe97c8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1894da50fe36470a9436c3339b564047 in term 0.
I20260924 15:18:05.749167 32454 leader_election.cc:302] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1894da50fe36470a9436c3339b564047, 5321774d284e418eac210a7735be4f56; no voters:
I20260924 15:18:05.749352 393 raft_consensus.cc:2781] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:18:05.749413 393 raft_consensus.cc:482] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:18:05.749462 393 raft_consensus.cc:3037] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:05.750175 393 raft_consensus.cc:504] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5321774d284e418eac210a7735be4f56" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 41615 } } peers { permanent_uuid: "1894da50fe36470a9436c3339b564047" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 39495 } } peers { permanent_uuid: "f6b44c830f794602b95125d0cfe97c8e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 40961 } }
I20260924 15:18:05.750314 393 leader_election.cc:288] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047 [CANDIDATE]: Term 1 election: Requested vote from peers 5321774d284e418eac210a7735be4f56 (127.31.12.66:41615), f6b44c830f794602b95125d0cfe97c8e (127.31.12.67:40961)
I20260924 15:18:05.750461 325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a46277b09054407a019a07090f7b2f9" candidate_uuid: "1894da50fe36470a9436c3339b564047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6b44c830f794602b95125d0cfe97c8e"
I20260924 15:18:05.750480 32657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a46277b09054407a019a07090f7b2f9" candidate_uuid: "1894da50fe36470a9436c3339b564047" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5321774d284e418eac210a7735be4f56"
I20260924 15:18:05.750545 32657 raft_consensus.cc:3037] T 6a46277b09054407a019a07090f7b2f9 P 5321774d284e418eac210a7735be4f56 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:05.750538 325 raft_consensus.cc:3037] T 6a46277b09054407a019a07090f7b2f9 P f6b44c830f794602b95125d0cfe97c8e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:05.751399 32657 raft_consensus.cc:2454] T 6a46277b09054407a019a07090f7b2f9 P 5321774d284e418eac210a7735be4f56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1894da50fe36470a9436c3339b564047 in term 1.
I20260924 15:18:05.751399 325 raft_consensus.cc:2454] T 6a46277b09054407a019a07090f7b2f9 P f6b44c830f794602b95125d0cfe97c8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1894da50fe36470a9436c3339b564047 in term 1.
I20260924 15:18:05.751582 32452 leader_election.cc:302] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1894da50fe36470a9436c3339b564047, f6b44c830f794602b95125d0cfe97c8e; no voters:
I20260924 15:18:05.751677 393 raft_consensus.cc:2781] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:05.751823 393 raft_consensus.cc:686] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047 [term 1 LEADER]: Becoming Leader. State: Replica: 1894da50fe36470a9436c3339b564047, State: Running, Role: LEADER
I20260924 15:18:05.751961 393 consensus_queue.cc:237] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5321774d284e418eac210a7735be4f56" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 41615 } } peers { permanent_uuid: "1894da50fe36470a9436c3339b564047" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 39495 } } peers { permanent_uuid: "f6b44c830f794602b95125d0cfe97c8e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 40961 } }
I20260924 15:18:05.752862 32373 catalog_manager.cc:5495] T 6a46277b09054407a019a07090f7b2f9 P 1894da50fe36470a9436c3339b564047 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1894da50fe36470a9436c3339b564047 (127.31.12.65). New cstate: current_term: 1 leader_uuid: "1894da50fe36470a9436c3339b564047" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5321774d284e418eac210a7735be4f56" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 41615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1894da50fe36470a9436c3339b564047" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 39495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f6b44c830f794602b95125d0cfe97c8e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 40961 } health_report { overall_health: UNKNOWN } } }
Sep 24 15:18:05 dist-test-slave-zwbb krb5kdc[32328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.65@KRBTEST.COM
I20260924 15:18:05.808302 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:05.800722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37318 (local address 127.31.12.65:39495)
0924 15:18:05.800801 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:05.800806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:05.800940 (+ 134us) server_negotiation.cc:407] Connection header received
0924 15:18:05.800967 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:05.800968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:05.800978 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:05.801031 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:05.801630 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.802245 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.803084 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.803288 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.803345 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:05.805771 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:05.805783 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:05.805783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:05.805789 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:05.807215 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:05.807541 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:05.807544 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:05.807545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:05.807607 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:05.807778 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:05.807779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:05.807779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:05.807918 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:05.808058 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:05.808088 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 15:18:05.808152 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 24 15:18:05 dist-test-slave-zwbb krb5kdc[32328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263085, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 15:18:05 dist-test-slave-zwbb krb5kdc[32328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263085, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.12.65@KRBTEST.COM
I20260924 15:18:05.831104 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:05.824192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.31.12.65:39495)
0924 15:18:05.824255 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:05.824257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:05.824376 (+ 119us) server_negotiation.cc:407] Connection header received
0924 15:18:05.824422 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:05.824422 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:05.824432 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:05.824480 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:05.824993 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.825527 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.826201 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.826382 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.826430 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:05.828628 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:05.828629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:05.828630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:05.828636 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:05.829966 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:05.830309 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:05.830310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:05.830310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:05.830361 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:05.830519 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:05.830520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:05.830520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:05.830607 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:05.830679 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:05.830887 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 15:18:05.830977 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 15:18:05 dist-test-slave-zwbb krb5kdc[32328](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 15:18:05 dist-test-slave-zwbb krb5kdc[32328](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263085, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.65@KRBTEST.COM
I20260924 15:18:05.853011 384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:05.846395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37338 (local address 127.31.12.65:39495)
0924 15:18:05.846459 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:05.846461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:05.846627 (+ 166us) server_negotiation.cc:407] Connection header received
0924 15:18:05.846657 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:05.846658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:05.846667 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:05.846723 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:05.847228 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.847686 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.848347 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:05.848519 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:05.848565 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:05.850800 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:05.850801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:05.850801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:05.850808 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:05.852170 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:05.852470 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:05.852470 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:05.852471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:05.852518 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:05.852663 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:05.852664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:05.852664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:05.852745 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:05.852831 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:05.852892 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 15:18:05.852947 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 15:18:05.853575 32480 tablet_service.cc:2908] Rejecting scan request for tablet 6a46277b09054407a019a07090f7b2f9: Uninitialized: clean time has not yet been initialized
I20260924 15:18:05.854380 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 32436
I20260924 15:18:05.860067 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 32572
I20260924 15:18:05.864638 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 32708
I20260924 15:18:05.869558 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 32341
2026-09-24T15:18:05Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2235 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:18:05 dist-test-slave-zwbb krb5kdc[410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:18:05 dist-test-slave-zwbb krb5kdc[410](info): set up 2 sockets
Sep 24 15:18:05 dist-test-slave-zwbb krb5kdc[410](info): commencing operation
krb5kdc: starting...
W20260924 15:18:06.906903 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:18:06 dist-test-slave-zwbb krb5kdc[410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:18:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:18:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:18:07.024627 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:34383
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:36531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:34383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:07.103637 423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:07.103870 423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:07.103921 423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:07.105110 423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:18:07.105180 423 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 15:18:07.105203 423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:07.105225 423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:07.105257 423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:18:07.105288 423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:18:07.106747 423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:36531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:34383
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:34383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:07.107013 423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:07.107230 423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:07.111754 428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:07.111748 431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:07.111753 429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:07.111778 423 server_base.cc:1034] running on GCE node
I20260924 15:18:07.112218 423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:07.112546 423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:07.113746 423 hybrid_clock.cc:648] HybridClock initialized: now 1790263087113722 us; error 38 us; skew 500 ppm
Sep 24 15:18:07 dist-test-slave-zwbb krb5kdc[410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263087, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:07.116078 423 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:18:07.116532 423 webserver.cc:460] Webserver started at http://127.31.12.126:37613/ using document root <none> and password file <none>
I20260924 15:18:07.116762 423 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:07.116827 423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:07.116976 423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:07.118047 423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "57fde51b60654080b094758aebd8005b"
format_stamp: "Formatted at 2026-09-24 15:18:07 on dist-test-slave-zwbb"
server_key: "0b2531fa1b008e8bae3dd50a92a6ff11"
server_key_iv: "5a3b9ed023a242ec0c903199978af724"
server_key_version: "encryptionkey@0"
I20260924 15:18:07.118404 423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "57fde51b60654080b094758aebd8005b"
format_stamp: "Formatted at 2026-09-24 15:18:07 on dist-test-slave-zwbb"
server_key: "0b2531fa1b008e8bae3dd50a92a6ff11"
server_key_iv: "5a3b9ed023a242ec0c903199978af724"
server_key_version: "encryptionkey@0"
I20260924 15:18:07.119668 423 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:18:07.120379 438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:07.120548 423 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:07.120649 423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "57fde51b60654080b094758aebd8005b"
format_stamp: "Formatted at 2026-09-24 15:18:07 on dist-test-slave-zwbb"
server_key: "0b2531fa1b008e8bae3dd50a92a6ff11"
server_key_iv: "5a3b9ed023a242ec0c903199978af724"
server_key_version: "encryptionkey@0"
I20260924 15:18:07.120721 423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:07.132431 423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:07.134330 423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:07.134500 423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:07.139459 423 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:34383
I20260924 15:18:07.139525 502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:34383 every 8 connection(s)
I20260924 15:18:07.139832 423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:18:07.140446 503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:07.142851 503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b: Bootstrap starting.
I20260924 15:18:07.143441 503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:07.143741 503 log.cc:824] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:07.144495 503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b: No bootstrap required, opened a new log
I20260924 15:18:07.146770 503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57fde51b60654080b094758aebd8005b" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 34383 } }
I20260924 15:18:07.146934 503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:07.147020 503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57fde51b60654080b094758aebd8005b, State: Initialized, Role: FOLLOWER
I20260924 15:18:07.147187 503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57fde51b60654080b094758aebd8005b" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 34383 } }
I20260924 15:18:07.147272 503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:18:07.147316 503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:18:07.147363 503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:07.147894 503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57fde51b60654080b094758aebd8005b" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 34383 } }
I20260924 15:18:07.148010 503 leader_election.cc:302] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57fde51b60654080b094758aebd8005b; no voters:
I20260924 15:18:07.148195 503 leader_election.cc:288] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:18:07.148252 506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:07.148448 503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:18:07.148531 506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b [term 1 LEADER]: Becoming Leader. State: Replica: 57fde51b60654080b094758aebd8005b, State: Running, Role: LEADER
I20260924 15:18:07.148631 506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57fde51b60654080b094758aebd8005b" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 34383 } }
I20260924 15:18:07.149000 508 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57fde51b60654080b094758aebd8005b. Latest consensus state: current_term: 1 leader_uuid: "57fde51b60654080b094758aebd8005b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57fde51b60654080b094758aebd8005b" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 34383 } } }
I20260924 15:18:07.149088 508 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:07.149081 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 423
I20260924 15:18:07.149183 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:18:07.149351 31793 external_mini_cluster.cc:1428] Setting key 210f1bd0312aa4a18417ff20b88cd53b
I20260924 15:18:07.149330 516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:18:07.149915 516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:18:07.150519 507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57fde51b60654080b094758aebd8005b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57fde51b60654080b094758aebd8005b" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 34383 } } }
I20260924 15:18:07.150666 507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:07.151471 516 catalog_manager.cc:1347] Generated new cluster ID: d5a970bf169b4cd4a34b2cec8a9ed13e
I20260924 15:18:07.151525 516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:18:07.155963 516 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:18:07.156442 516 catalog_manager.cc:1504] Loading token signing keys...
Sep 24 15:18:07 dist-test-slave-zwbb krb5kdc[410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:07.161829 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.150043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51398 (local address 127.31.12.126:34383)
0924 15:18:07.150603 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:07.150613 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:07.150896 (+ 283us) server_negotiation.cc:407] Connection header received
0924 15:18:07.151033 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:07.151040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:07.151098 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:07.151274 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:07.152019 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.153432 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.154558 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.155170 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.155275 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.158151 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:07.158155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:07.158156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:07.158168 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:07.160184 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:07.160588 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:07.160592 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:07.160592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:07.160703 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:07.160908 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:07.160909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:07.160909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:07.161184 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:07.161336 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:07.161456 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 15:18:07.161609 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":190,"threads_started":1}
I20260924 15:18:07.171517 516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b: Generated new TSK 0
I20260924 15:18:07.171702 516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
I20260924 15:18:07.201608 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:34383
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:36531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:07.281863 530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:07.282043 530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:07.282073 530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:07.283156 530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:07.283191 530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:07.283212 530 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:18:07.283229 530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:18:07.284547 530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:36531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.12.126:34383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:07.284725 530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:07.284914 530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:18:07.289752 530 server_base.cc:1034] running on GCE node
W20260924 15:18:07.289718 536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:07.289763 535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:07.289724 538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:07.290165 530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:07.290408 530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:07.291550 530 hybrid_clock.cc:648] HybridClock initialized: now 1790263087291540 us; error 28 us; skew 500 ppm
Sep 24 15:18:07 dist-test-slave-zwbb krb5kdc[410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263087, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:07.293972 530 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:18:07.294376 530 webserver.cc:460] Webserver started at http://127.31.12.65:42397/ using document root <none> and password file <none>
I20260924 15:18:07.294618 530 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:07.294687 530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:07.294836 530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:07.295840 530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "fdc999b61b7a4db1802dc9279840129c"
format_stamp: "Formatted at 2026-09-24 15:18:07 on dist-test-slave-zwbb"
server_key: "676642173c2164fa45a5e3126b38cfbd"
server_key_iv: "4987bbc50796a440b3a55ab01dd4eb95"
server_key_version: "encryptionkey@0"
I20260924 15:18:07.296200 530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "fdc999b61b7a4db1802dc9279840129c"
format_stamp: "Formatted at 2026-09-24 15:18:07 on dist-test-slave-zwbb"
server_key: "676642173c2164fa45a5e3126b38cfbd"
server_key_iv: "4987bbc50796a440b3a55ab01dd4eb95"
server_key_version: "encryptionkey@0"
I20260924 15:18:07.297536 530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:18:07.298255 545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:07.298421 530 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:07.298527 530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "fdc999b61b7a4db1802dc9279840129c"
format_stamp: "Formatted at 2026-09-24 15:18:07 on dist-test-slave-zwbb"
server_key: "676642173c2164fa45a5e3126b38cfbd"
server_key_iv: "4987bbc50796a440b3a55ab01dd4eb95"
server_key_version: "encryptionkey@0"
I20260924 15:18:07.298611 530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:07.331040 530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:07.332891 530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:07.333060 530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:07.333668 530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:07.333734 530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:07.333778 530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:07.333806 530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:18:07 dist-test-slave-zwbb krb5kdc[410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263087, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:07.340788 530 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:37673
I20260924 15:18:07.340869 663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:37673 every 8 connection(s)
I20260924 15:18:07.341216 530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:18:07.343540 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.334248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:41359 (local address 127.31.12.126:34383)
0924 15:18:07.334302 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:07.334304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:07.334545 (+ 241us) server_negotiation.cc:407] Connection header received
0924 15:18:07.335089 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:07.335091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:07.335104 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:07.335167 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:07.336034 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.336730 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.338148 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.338366 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.338460 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.340859 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:07.340861 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:07.340861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:07.340868 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:07.342396 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:07.342757 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:07.342758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:07.342758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:07.342807 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:07.343025 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:07.343026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:07.343026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:07.343126 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:07.343176 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:07.343395 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 15:18:07.343468 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:18:07.343703 554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.334254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:34383 (local address 127.31.12.65:41359)
0924 15:18:07.334424 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:07.334461 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:07.334988 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:07.335243 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:07.335247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:07.335626 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:07.335920 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.335951 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.336829 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.336830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.338060 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.338064 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.338494 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.338495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.338623 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.339142 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:07.339150 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:07.340757 (+ 1607us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:07.342471 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:07.342476 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:07.342477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:07.342687 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:07.342863 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:07.342863 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:07.342863 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:07.342970 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:07.343181 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:07.343186 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:07.343295 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 15:18:07.343373 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 15:18:07.343491 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":83,"threads_started":1}
I20260924 15:18:07.344209 664 heartbeater.cc:347] Connected to a master server at 127.31.12.126:34383
I20260924 15:18:07.344313 664 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:07.344485 664 heartbeater.cc:511] Master 127.31.12.126:34383 requested a full tablet report, sending...
I20260924 15:18:07.345070 454 ts_manager.cc:194] Registered new tserver with Master: fdc999b61b7a4db1802dc9279840129c (127.31.12.65:37673)
I20260924 15:18:07.345809 454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:41359
I20260924 15:18:07.346700 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 530
I20260924 15:18:07.346803 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:18:07.346967 31793 external_mini_cluster.cc:1428] Setting key 4d4c683d160b4ed06f8fc9384112e597
I20260924 15:18:07.351243 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.345212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:34383 (local address 127.31.12.65:57605)
0924 15:18:07.345345 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:07.345355 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:07.345435 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:07.345582 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:07.345583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:07.345756 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:07.345802 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.345803 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.346516 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.346517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.347176 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.347179 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.347537 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.347538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.347615 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.348055 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:07.348060 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:07.348381 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:07.350343 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:07.350344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:07.350345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:07.350517 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:07.350716 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:07.350717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:07.350717 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:07.350767 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:07.351023 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:07.351023 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:07.351058 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 15:18:07.351108 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 15:18:07.351180 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260924 15:18:07.351351 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.345256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:57605 (local address 127.31.12.126:34383)
0924 15:18:07.345290 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:07.345292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:07.345421 (+ 129us) server_negotiation.cc:407] Connection header received
0924 15:18:07.345511 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:07.345512 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:07.345521 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:07.345577 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:07.345880 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.346409 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.347240 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.347452 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.347535 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.348484 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:07.348486 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:07.348486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:07.348493 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:07.350264 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:07.350590 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:07.350591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:07.350592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:07.350653 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:07.350826 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:07.350827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:07.350827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:07.350962 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:07.351029 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:07.351131 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 15:18:07.351213 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
WARNING: no policy specified for kudu/127.31.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
I20260924 15:18:07.373646 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:34383
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:36531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:07.453397 673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:07.453611 673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:07.453635 673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:07.454788 673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:07.454828 673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:07.454849 673 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:18:07.454909 673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:18:07.456310 673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:36531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.12.126:34383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:07.456561 673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:07.456794 673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:07.461768 679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:07.461777 673 server_base.cc:1034] running on GCE node
W20260924 15:18:07.461760 681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:07.461760 678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:07.462186 673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:07.462381 673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:07.463521 673 hybrid_clock.cc:648] HybridClock initialized: now 1790263087463501 us; error 36 us; skew 500 ppm
Sep 24 15:18:07 dist-test-slave-zwbb krb5kdc[410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263087, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:07.465865 673 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:18:07.466324 673 webserver.cc:460] Webserver started at http://127.31.12.66:35691/ using document root <none> and password file <none>
I20260924 15:18:07.466588 673 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:07.466658 673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:07.466802 673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:07.467813 673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "0f7e7edf1af84b6bb40550d953467e9e"
format_stamp: "Formatted at 2026-09-24 15:18:07 on dist-test-slave-zwbb"
server_key: "01e9eaf47c7d8842fdef5e214ea1b944"
server_key_iv: "f542d514dab545836d9deb49000ef3cb"
server_key_version: "encryptionkey@0"
I20260924 15:18:07.468164 673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "0f7e7edf1af84b6bb40550d953467e9e"
format_stamp: "Formatted at 2026-09-24 15:18:07 on dist-test-slave-zwbb"
server_key: "01e9eaf47c7d8842fdef5e214ea1b944"
server_key_iv: "f542d514dab545836d9deb49000ef3cb"
server_key_version: "encryptionkey@0"
I20260924 15:18:07.469485 673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:18:07.470221 688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:07.470412 673 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:07.470532 673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "0f7e7edf1af84b6bb40550d953467e9e"
format_stamp: "Formatted at 2026-09-24 15:18:07 on dist-test-slave-zwbb"
server_key: "01e9eaf47c7d8842fdef5e214ea1b944"
server_key_iv: "f542d514dab545836d9deb49000ef3cb"
server_key_version: "encryptionkey@0"
I20260924 15:18:07.470615 673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:07.499562 673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:07.501613 673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:07.501766 673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:07.502373 673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:07.502434 673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:07.502465 673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:07.502489 673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:18:07 dist-test-slave-zwbb krb5kdc[410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263087, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:07.509672 673 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:33859
I20260924 15:18:07.510020 673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:18:07.510502 805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:33859 every 8 connection(s)
I20260924 15:18:07.512666 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.502792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:55171 (local address 127.31.12.126:34383)
0924 15:18:07.502868 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:07.502871 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:07.503098 (+ 227us) server_negotiation.cc:407] Connection header received
0924 15:18:07.503593 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:07.503594 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:07.503610 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:07.503696 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:07.504380 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.504916 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.506007 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.506222 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.506308 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.509706 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:07.509708 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:07.509708 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:07.509715 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:07.511243 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:07.511671 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:07.511672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:07.511673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:07.511732 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:07.512007 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:07.512008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:07.512008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:07.512134 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:07.512205 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:07.512486 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 15:18:07.512576 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:18:07.512850 697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.502772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:34383 (local address 127.31.12.66:55171)
0924 15:18:07.502962 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:07.503002 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:07.503497 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:07.503782 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:07.503786 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:07.504088 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:07.504278 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.504284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.505028 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.505030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.505911 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.505913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.506319 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.506320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.506451 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.506943 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:07.506954 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:07.509596 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:07.511345 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:07.511352 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:07.511355 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:07.511579 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:07.511802 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:07.511803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:07.511803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:07.511927 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:07.512210 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:07.512216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:07.512351 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 15:18:07.512467 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 15:18:07.512623 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":117,"threads_started":1}
I20260924 15:18:07.513329 806 heartbeater.cc:347] Connected to a master server at 127.31.12.126:34383
I20260924 15:18:07.513417 806 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:07.513634 806 heartbeater.cc:511] Master 127.31.12.126:34383 requested a full tablet report, sending...
I20260924 15:18:07.514077 455 ts_manager.cc:194] Registered new tserver with Master: 0f7e7edf1af84b6bb40550d953467e9e (127.31.12.66:33859)
I20260924 15:18:07.514652 455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:55171
I20260924 15:18:07.519515 812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.514441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:34383 (local address 127.31.12.66:44759)
0924 15:18:07.514568 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:07.514578 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:07.514629 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:07.514776 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:07.514776 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:07.514974 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:07.515033 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.515034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.515790 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.515791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.516287 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.516288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.516585 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.516585 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.516640 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.516981 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:07.516985 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:07.517247 (+ 262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:07.518718 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:07.518719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:07.518719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:07.518889 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:07.519049 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:07.519049 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:07.519050 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:07.519092 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:07.519293 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:07.519294 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:07.519327 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 15:18:07.519380 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 15:18:07.519449 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260924 15:18:07.519518 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 673
I20260924 15:18:07.519601 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:18:07.519611 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.514489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:44759 (local address 127.31.12.126:34383)
0924 15:18:07.514523 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:07.514525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:07.514623 (+ 98us) server_negotiation.cc:407] Connection header received
0924 15:18:07.514695 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:07.514696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:07.514703 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:07.514772 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:07.515102 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.515709 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.516355 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.516526 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.516571 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.517312 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:07.517313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:07.517313 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:07.517319 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:07.518657 (+ 1338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:07.518960 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:07.518961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:07.518961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:07.519006 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:07.519123 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:07.519124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:07.519124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:07.519241 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:07.519287 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:07.519435 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 15:18:07.519514 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 15:18:07.519769 31793 external_mini_cluster.cc:1428] Setting key 2bc3c0de5657a268d7c5740b648b936e
WARNING: no policy specified for kudu/127.31.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
I20260924 15:18:07.543937 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:34383
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:36531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:07.624260 815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:07.624483 815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:07.624521 815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:07.625733 815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:07.625800 815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:07.625835 815 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:18:07.625875 815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:18:07.627277 815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:36531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.12.126:34383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:07.627549 815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:07.627775 815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:07.632844 821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:07.632999 823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:07.633038 815 server_base.cc:1034] running on GCE node
W20260924 15:18:07.633005 820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:07.633407 815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:07.633694 815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:07.634859 815 hybrid_clock.cc:648] HybridClock initialized: now 1790263087634848 us; error 24 us; skew 500 ppm
Sep 24 15:18:07 dist-test-slave-zwbb krb5kdc[410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263087, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:07.637316 815 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:18:07.637899 815 webserver.cc:460] Webserver started at http://127.31.12.67:44595/ using document root <none> and password file <none>
I20260924 15:18:07.638206 815 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:07.638273 815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:07.638418 815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:07.639950 815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "bef4c4e4b06e4113b313c8df1aac9ccb"
format_stamp: "Formatted at 2026-09-24 15:18:07 on dist-test-slave-zwbb"
server_key: "0d18f412899def3ae45d53ba8f0a80f8"
server_key_iv: "fe5b0075c335d3bab605876130f6d878"
server_key_version: "encryptionkey@0"
I20260924 15:18:07.640357 815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "bef4c4e4b06e4113b313c8df1aac9ccb"
format_stamp: "Formatted at 2026-09-24 15:18:07 on dist-test-slave-zwbb"
server_key: "0d18f412899def3ae45d53ba8f0a80f8"
server_key_iv: "fe5b0075c335d3bab605876130f6d878"
server_key_version: "encryptionkey@0"
I20260924 15:18:07.641716 815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260924 15:18:07.642506 830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:07.642704 815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:07.642805 815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "bef4c4e4b06e4113b313c8df1aac9ccb"
format_stamp: "Formatted at 2026-09-24 15:18:07 on dist-test-slave-zwbb"
server_key: "0d18f412899def3ae45d53ba8f0a80f8"
server_key_iv: "fe5b0075c335d3bab605876130f6d878"
server_key_version: "encryptionkey@0"
I20260924 15:18:07.642894 815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:07.655483 815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:07.657204 815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:07.657361 815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:07.657999 815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:07.658047 815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:07.658085 815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:07.658099 815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:18:07 dist-test-slave-zwbb krb5kdc[410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263087, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:07.665522 815 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:41173
I20260924 15:18:07.665628 947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:41173 every 8 connection(s)
I20260924 15:18:07.665951 815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:18:07.668324 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.658442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:60701 (local address 127.31.12.126:34383)
0924 15:18:07.658525 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:07.658528 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:07.658794 (+ 266us) server_negotiation.cc:407] Connection header received
0924 15:18:07.659571 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:07.659573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:07.659588 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:07.659671 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:07.660603 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.661141 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.662811 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.663023 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.663100 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.665682 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:07.665683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:07.665683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:07.665691 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:07.667189 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:07.667564 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:07.667565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:07.667565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:07.667624 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:07.667837 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:07.667837 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:07.667837 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:07.667958 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:07.668020 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:07.668203 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 15:18:07.668264 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 15:18:07.668534 839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.658424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:34383 (local address 127.31.12.67:60701)
0924 15:18:07.658643 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:07.658697 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:07.659397 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:07.659764 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:07.659770 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:07.660198 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:07.660498 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.660506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.661254 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.661255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.662720 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.662725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.663101 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.663102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.663264 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.663778 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:07.663790 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:07.665595 (+ 1805us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:07.667257 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:07.667263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:07.667265 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:07.667477 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:07.667689 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:07.667689 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:07.667690 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:07.667780 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:07.668012 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:07.668015 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:07.668114 (+ 99us) client_negotiation.cc:769] Sending connection context
0924 15:18:07.668188 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 15:18:07.668298 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":108,"threads_started":1}
I20260924 15:18:07.668529 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 815
I20260924 15:18:07.668607 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:18:07.668790 31793 external_mini_cluster.cc:1428] Setting key 2732de38a3b7c510ce777990a520aad2
I20260924 15:18:07.668987 948 heartbeater.cc:347] Connected to a master server at 127.31.12.126:34383
I20260924 15:18:07.669063 948 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:07.669252 948 heartbeater.cc:511] Master 127.31.12.126:34383 requested a full tablet report, sending...
I20260924 15:18:07.669629 455 ts_manager.cc:194] Registered new tserver with Master: bef4c4e4b06e4113b313c8df1aac9ccb (127.31.12.67:41173)
I20260924 15:18:07.670022 455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:60701
I20260924 15:18:07.675019 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.670108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:46475 (local address 127.31.12.126:34383)
0924 15:18:07.670152 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:07.670153 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:07.670286 (+ 133us) server_negotiation.cc:407] Connection header received
0924 15:18:07.670346 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:07.670346 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:07.670356 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:07.670408 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:07.670730 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.671167 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.671830 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.672015 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.672068 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.672782 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:07.672783 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:07.672783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:07.672790 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:07.674140 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:07.674487 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:07.674488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:07.674488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:07.674536 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:07.674682 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:07.674682 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:07.674682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:07.674763 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:07.674807 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:07.674906 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 15:18:07.674962 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:18:07.675038 954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.670045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:34383 (local address 127.31.12.67:46475)
0924 15:18:07.670200 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:07.670213 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:07.670287 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:07.670408 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:07.670409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:07.670603 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:07.670658 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.670659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.671269 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.671270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.671766 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.671767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.672069 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.672070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.672124 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.672454 (+ 330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:07.672458 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:07.672709 (+ 251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:07.674207 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:07.674208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:07.674208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:07.674422 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:07.674594 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:07.674594 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:07.674595 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:07.674644 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:07.674810 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:07.674811 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:07.674843 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 15:18:07.674892 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 15:18:07.674976 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":75,"threads_started":1}
I20260924 15:18:07.680401 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 15:18:07 dist-test-slave-zwbb krb5kdc[410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263087, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 15:18:07 dist-test-slave-zwbb krb5kdc[410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263087, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:07.704421 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.696732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51414 (local address 127.31.12.126:34383)
0924 15:18:07.696822 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:07.696824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:07.696832 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:18:07.696858 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:07.696859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:07.696868 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:07.696921 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:07.697699 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.698203 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.698925 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.699129 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.699183 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.701724 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:07.701726 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:07.701726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:07.701733 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:07.703414 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:07.703752 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:07.703753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:07.703753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:07.703808 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:07.703959 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:07.703959 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:07.703959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:07.704061 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:07.704138 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:07.704224 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 15:18:07.704287 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 15:18:07.706476 455 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:18:07.708207 455 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:18:07.715296 965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.711132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:41173 (local address 127.0.0.1:47524)
0924 15:18:07.711531 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:07.711541 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:07.711595 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:07.711838 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:07.711839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:07.711841 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:07.711888 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.711889 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.712746 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.712747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.713884 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.713887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.714922 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.714923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.715018 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.715034 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 15:18:07.715133 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 15:18:07.715197 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":93,"threads_started":1}
I20260924 15:18:07.715327 970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.711353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47524 (local address 127.31.12.67:41173)
0924 15:18:07.711526 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:07.711531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:07.711649 (+ 118us) server_negotiation.cc:407] Connection header received
0924 15:18:07.711693 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:07.711695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:07.711711 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:07.711780 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:07.711939 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.712660 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.713959 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.714831 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.714923 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.714992 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:07.715164 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 15:18:07.715204 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260924 15:18:07.715443 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.710904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:33859 (local address 127.0.0.1:53248)
0924 15:18:07.711137 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:07.711184 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:07.711247 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:07.711743 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:07.711747 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:07.711749 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:07.711820 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.711822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.713312 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.713313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.714268 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.714270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.715205 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.715206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.715280 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.715284 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:18:07.715314 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 15:18:07.715356 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":76,"threads_started":1}
I20260924 15:18:07.715734 968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.710983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:37673 (local address 127.0.0.1:49658)
0924 15:18:07.711278 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:07.711291 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:07.711348 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:07.711809 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:07.711810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:07.711812 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:07.711901 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.711903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.713394 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.713395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.714592 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.714595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.715539 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.715540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.715602 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.715604 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:18:07.715634 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 15:18:07.715666 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":95,"threads_started":1}
I20260924 15:18:07.715751 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.711029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53248 (local address 127.31.12.66:33859)
0924 15:18:07.711425 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:07.711431 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:07.711443 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:18:07.711488 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:07.711491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:07.711509 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:07.711599 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:07.711904 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.713192 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.714359 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.715102 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.715524 (+ 422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.715588 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:07.715614 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 15:18:07.715656 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":89,"threads_started":1}
I20260924 15:18:07.716113 969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.711194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49658 (local address 127.31.12.65:37673)
0924 15:18:07.711617 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:07.711621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:07.711632 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:18:07.711672 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:07.711675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:07.711689 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:07.711992 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:07.712062 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.713265 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.714690 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.715432 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.715878 (+ 446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.715950 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:07.715975 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 15:18:07.716012 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":92,"threads_started":1}
I20260924 15:18:07.716471 880 tablet_service.cc:1467] Processing CreateTablet for tablet 8692755dd3c441fdaecae7b98f2d9524 (DEFAULT_TABLE table=test-table [id=bd41f584747745588f6e65d261a75b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:07.716560 738 tablet_service.cc:1467] Processing CreateTablet for tablet 8692755dd3c441fdaecae7b98f2d9524 (DEFAULT_TABLE table=test-table [id=bd41f584747745588f6e65d261a75b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:07.716702 596 tablet_service.cc:1467] Processing CreateTablet for tablet 8692755dd3c441fdaecae7b98f2d9524 (DEFAULT_TABLE table=test-table [id=bd41f584747745588f6e65d261a75b8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:07.716856 880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8692755dd3c441fdaecae7b98f2d9524. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:07.716928 738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8692755dd3c441fdaecae7b98f2d9524. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:07.717015 596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8692755dd3c441fdaecae7b98f2d9524. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:07.720911 973 tablet_bootstrap.cc:492] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c: Bootstrap starting.
I20260924 15:18:07.720922 972 tablet_bootstrap.cc:492] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e: Bootstrap starting.
I20260924 15:18:07.721129 971 tablet_bootstrap.cc:492] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb: Bootstrap starting.
I20260924 15:18:07.721637 973 tablet_bootstrap.cc:654] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:07.721637 972 tablet_bootstrap.cc:654] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:07.722000 973 log.cc:824] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:07.722000 972 log.cc:824] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:07.722097 971 tablet_bootstrap.cc:654] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:07.722429 971 log.cc:824] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:07.722832 972 tablet_bootstrap.cc:492] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e: No bootstrap required, opened a new log
I20260924 15:18:07.722937 972 ts_tablet_manager.cc:1397] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:18:07.723039 973 tablet_bootstrap.cc:492] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c: No bootstrap required, opened a new log
I20260924 15:18:07.723115 973 ts_tablet_manager.cc:1397] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:18:07.723431 971 tablet_bootstrap.cc:492] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb: No bootstrap required, opened a new log
I20260924 15:18:07.723497 971 ts_tablet_manager.cc:1397] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:18:07.724666 972 raft_consensus.cc:348] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } } peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } }
I20260924 15:18:07.724815 972 raft_consensus.cc:374] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:07.724874 972 raft_consensus.cc:729] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f7e7edf1af84b6bb40550d953467e9e, State: Initialized, Role: FOLLOWER
I20260924 15:18:07.725049 972 consensus_queue.cc:260] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } } peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } }
I20260924 15:18:07.725361 806 heartbeater.cc:503] Master 127.31.12.126:34383 was elected leader, sending a full tablet report...
I20260924 15:18:07.725302 973 raft_consensus.cc:348] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } } peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } }
I20260924 15:18:07.725493 973 raft_consensus.cc:374] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:07.725548 973 raft_consensus.cc:729] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc999b61b7a4db1802dc9279840129c, State: Initialized, Role: FOLLOWER
I20260924 15:18:07.725662 973 consensus_queue.cc:260] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } } peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } }
I20260924 15:18:07.725716 972 ts_tablet_manager.cc:1428] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:18:07.725939 973 ts_tablet_manager.cc:1428] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:18:07.725999 664 heartbeater.cc:503] Master 127.31.12.126:34383 was elected leader, sending a full tablet report...
I20260924 15:18:07.726188 971 raft_consensus.cc:348] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } } peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } }
I20260924 15:18:07.726342 971 raft_consensus.cc:374] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:07.726382 971 raft_consensus.cc:729] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bef4c4e4b06e4113b313c8df1aac9ccb, State: Initialized, Role: FOLLOWER
I20260924 15:18:07.726485 971 consensus_queue.cc:260] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } } peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } }
I20260924 15:18:07.726773 971 ts_tablet_manager.cc:1428] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 15:18:07.726846 948 heartbeater.cc:503] Master 127.31.12.126:34383 was elected leader, sending a full tablet report...
I20260924 15:18:07.731514 978 raft_consensus.cc:482] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:18:07.731601 978 raft_consensus.cc:504] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } } peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } }
I20260924 15:18:07.731940 978 leader_election.cc:288] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f7e7edf1af84b6bb40550d953467e9e (127.31.12.66:33859), bef4c4e4b06e4113b313c8df1aac9ccb (127.31.12.67:41173)
I20260924 15:18:07.734943 554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.732153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:33859 (local address 127.31.12.65:45067)
0924 15:18:07.732218 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:07.732232 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:07.732292 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:07.732438 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:07.732439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:07.732441 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:07.732520 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.732522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.733285 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.733286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.734111 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.734113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.734672 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.734673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.734767 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.734770 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:18:07.734835 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 15:18:07.734876 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 15:18:07.734943 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.732226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:45067 (local address 127.31.12.66:33859)
0924 15:18:07.732294 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:07.732296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:07.732307 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:18:07.732342 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:07.732343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:07.732353 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:07.732420 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:07.732586 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.733204 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.734216 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.734577 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.734672 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.734681 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:07.734840 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 15:18:07.734876 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 15:18:07.735212 970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.732468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:47889 (local address 127.31.12.67:41173)
0924 15:18:07.732516 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:07.732519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:07.732619 (+ 100us) server_negotiation.cc:407] Connection header received
0924 15:18:07.732679 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:07.732680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:07.732689 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:07.732742 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:07.732953 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.733382 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.734410 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.734789 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.734882 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.734890 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:07.735107 (+ 217us) server_negotiation.cc:299] Negotiation successful
0924 15:18:07.735156 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:18:07.735272 759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8692755dd3c441fdaecae7b98f2d9524" candidate_uuid: "fdc999b61b7a4db1802dc9279840129c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f7e7edf1af84b6bb40550d953467e9e" is_pre_election: true
I20260924 15:18:07.735419 759 raft_consensus.cc:2454] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc999b61b7a4db1802dc9279840129c in term 0.
I20260924 15:18:07.735450 980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.732262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:41173 (local address 127.31.12.65:47889)
0924 15:18:07.732565 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:07.732575 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:07.732629 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:07.732782 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:07.732783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:07.732789 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:07.732862 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.732863 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.733579 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.733580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.734299 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.734301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.734885 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.734886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.735006 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.735010 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:18:07.735321 (+ 311us) client_negotiation.cc:240] Negotiation successful
0924 15:18:07.735366 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":101,"threads_started":1}
I20260924 15:18:07.735749 546 leader_election.cc:302] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f7e7edf1af84b6bb40550d953467e9e, fdc999b61b7a4db1802dc9279840129c; no voters:
I20260924 15:18:07.735785 901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8692755dd3c441fdaecae7b98f2d9524" candidate_uuid: "fdc999b61b7a4db1802dc9279840129c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" is_pre_election: true
I20260924 15:18:07.735884 978 raft_consensus.cc:2781] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:18:07.735908 901 raft_consensus.cc:2454] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fdc999b61b7a4db1802dc9279840129c in term 0.
I20260924 15:18:07.735950 978 raft_consensus.cc:482] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:18:07.735987 978 raft_consensus.cc:3037] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:07.736678 978 raft_consensus.cc:504] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } } peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } }
I20260924 15:18:07.736815 978 leader_election.cc:288] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [CANDIDATE]: Term 1 election: Requested vote from peers 0f7e7edf1af84b6bb40550d953467e9e (127.31.12.66:33859), bef4c4e4b06e4113b313c8df1aac9ccb (127.31.12.67:41173)
I20260924 15:18:07.737000 759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8692755dd3c441fdaecae7b98f2d9524" candidate_uuid: "fdc999b61b7a4db1802dc9279840129c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f7e7edf1af84b6bb40550d953467e9e"
I20260924 15:18:07.737010 901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8692755dd3c441fdaecae7b98f2d9524" candidate_uuid: "fdc999b61b7a4db1802dc9279840129c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb"
I20260924 15:18:07.737114 759 raft_consensus.cc:3037] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:07.737112 901 raft_consensus.cc:3037] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:07.737860 759 raft_consensus.cc:2454] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc999b61b7a4db1802dc9279840129c in term 1.
I20260924 15:18:07.737913 901 raft_consensus.cc:2454] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fdc999b61b7a4db1802dc9279840129c in term 1.
I20260924 15:18:07.738081 546 leader_election.cc:302] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bef4c4e4b06e4113b313c8df1aac9ccb, fdc999b61b7a4db1802dc9279840129c; no voters:
I20260924 15:18:07.738190 978 raft_consensus.cc:2781] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:07.738335 978 raft_consensus.cc:686] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [term 1 LEADER]: Becoming Leader. State: Replica: fdc999b61b7a4db1802dc9279840129c, State: Running, Role: LEADER
I20260924 15:18:07.738437 978 consensus_queue.cc:237] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } } peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } }
I20260924 15:18:07.739223 454 catalog_manager.cc:5495] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c reported cstate change: term changed from 0 to 1, leader changed from <none> to fdc999b61b7a4db1802dc9279840129c (127.31.12.65). New cstate: current_term: 1 leader_uuid: "fdc999b61b7a4db1802dc9279840129c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } health_report { overall_health: HEALTHY } } }
I20260924 15:18:07.752445 969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.749600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49664 (local address 127.31.12.65:37673)
0924 15:18:07.749663 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:07.749666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:07.749699 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:18:07.749818 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:07.749819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:07.749843 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:07.749934 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:18:07.750079 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.750601 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.751398 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.751594 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.751671 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.751868 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:18:07.751968 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:18:07.752059 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:07.752118 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 15:18:07.752145 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:18:07.754297 901 raft_consensus.cc:1261] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb [term 1 FOLLOWER]: Refusing update from remote peer fdc999b61b7a4db1802dc9279840129c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:07.754330 759 raft_consensus.cc:1261] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e [term 1 FOLLOWER]: Refusing update from remote peer fdc999b61b7a4db1802dc9279840129c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:07.754698 981 consensus_queue.cc:1035] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [LEADER]: Connected to new peer: Peer: permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:07.754863 978 consensus_queue.cc:1035] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:07.757613 985 mvcc.cc:205] Tried to move back new op lower bound from 7332917607438704640 to 7332917607377240064. Current Snapshot: MvccSnapshot[applied={T|T < 7332917607438704640}]
I20260924 15:18:07.758504 986 mvcc.cc:205] Tried to move back new op lower bound from 7332917607438704640 to 7332917607377240064. Current Snapshot: MvccSnapshot[applied={T|T < 7332917607438704640 or (T in {7332917607438704640})}]
I20260924 15:18:07.761251 455 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:18:07.761356 455 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:18:07.762192 455 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b: Sending DeleteTablet for 3 replicas of tablet 8692755dd3c441fdaecae7b98f2d9524
I20260924 15:18:07.762537 596 tablet_service.cc:1514] Processing DeleteTablet for tablet 8692755dd3c441fdaecae7b98f2d9524 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:07 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:49658
I20260924 15:18:07.762617 880 tablet_service.cc:1514] Processing DeleteTablet for tablet 8692755dd3c441fdaecae7b98f2d9524 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:07 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:47524
I20260924 15:18:07.762746 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 8692755dd3c441fdaecae7b98f2d9524 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:07 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:53248
I20260924 15:18:07.762737 455 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:18:07.762879 455 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:18:07.762964 995 tablet_replica.cc:330] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e: stopping tablet replica
I20260924 15:18:07.763068 995 raft_consensus.cc:2229] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:18:07.763170 995 raft_consensus.cc:2258] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:07.763371 993 tablet_replica.cc:330] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c: stopping tablet replica
I20260924 15:18:07.763520 993 raft_consensus.cc:2229] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:18:07.763613 995 ts_tablet_manager.cc:1905] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:07.763753 993 raft_consensus.cc:2258] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:07.764190 596 tablet_service.cc:1467] Processing CreateTablet for tablet 8e23f5081c4f44ebbb9c4a60a8d301af (DEFAULT_TABLE table=test-table [id=5bbea4071bfd4037bc3548a5c89c4612]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:07.764307 596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e23f5081c4f44ebbb9c4a60a8d301af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:07.764292 880 tablet_service.cc:1467] Processing CreateTablet for tablet 8e23f5081c4f44ebbb9c4a60a8d301af (DEFAULT_TABLE table=test-table [id=5bbea4071bfd4037bc3548a5c89c4612]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:07.764424 880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e23f5081c4f44ebbb9c4a60a8d301af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:07.765417 973 tablet_bootstrap.cc:492] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c: Bootstrap starting.
I20260924 15:18:07.765491 738 tablet_service.cc:1467] Processing CreateTablet for tablet 8e23f5081c4f44ebbb9c4a60a8d301af (DEFAULT_TABLE table=test-table [id=5bbea4071bfd4037bc3548a5c89c4612]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:07.765591 738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e23f5081c4f44ebbb9c4a60a8d301af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:07.765805 971 tablet_bootstrap.cc:492] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb: Bootstrap starting.
I20260924 15:18:07.765846 973 tablet_bootstrap.cc:654] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:07.766343 971 tablet_bootstrap.cc:654] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:07.766969 972 tablet_bootstrap.cc:492] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e: Bootstrap starting.
I20260924 15:18:07.766983 971 tablet_bootstrap.cc:492] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb: No bootstrap required, opened a new log
I20260924 15:18:07.767031 971 ts_tablet_manager.cc:1397] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 15:18:07.767197 971 raft_consensus.cc:348] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } }
I20260924 15:18:07.767272 971 raft_consensus.cc:374] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:07.767297 971 raft_consensus.cc:729] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bef4c4e4b06e4113b313c8df1aac9ccb, State: Initialized, Role: FOLLOWER
I20260924 15:18:07.767355 971 consensus_queue.cc:260] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } }
I20260924 15:18:07.767390 972 tablet_bootstrap.cc:654] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:07.767458 971 ts_tablet_manager.cc:1428] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:07.768324 995 ts_tablet_manager.cc:1918] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:18:07.768404 995 log.cc:1196] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/8692755dd3c441fdaecae7b98f2d9524
I20260924 15:18:07.768674 995 ts_tablet_manager.cc:1939] T 8692755dd3c441fdaecae7b98f2d9524 P 0f7e7edf1af84b6bb40550d953467e9e: Deleting consensus metadata
I20260924 15:18:07.768702 993 ts_tablet_manager.cc:1905] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:07.769213 439 catalog_manager.cc:4841] TS 0f7e7edf1af84b6bb40550d953467e9e (127.31.12.66:33859): tablet 8692755dd3c441fdaecae7b98f2d9524 (table test-table [id=bd41f584747745588f6e65d261a75b8a]) successfully deleted
I20260924 15:18:07.770203 993 ts_tablet_manager.cc:1918] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:18:07.770288 993 log.cc:1196] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-0/wal/wals/8692755dd3c441fdaecae7b98f2d9524
I20260924 15:18:07.770574 993 ts_tablet_manager.cc:1939] T 8692755dd3c441fdaecae7b98f2d9524 P fdc999b61b7a4db1802dc9279840129c: Deleting consensus metadata
I20260924 15:18:07.770895 994 tablet_replica.cc:330] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb: stopping tablet replica
I20260924 15:18:07.770975 440 catalog_manager.cc:4841] TS fdc999b61b7a4db1802dc9279840129c (127.31.12.65:37673): tablet 8692755dd3c441fdaecae7b98f2d9524 (table test-table [id=bd41f584747745588f6e65d261a75b8a]) successfully deleted
I20260924 15:18:07.771019 994 raft_consensus.cc:2229] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:18:07.771117 994 raft_consensus.cc:2258] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:07.771525 994 ts_tablet_manager.cc:1905] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:07.771580 973 tablet_bootstrap.cc:492] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c: No bootstrap required, opened a new log
I20260924 15:18:07.771631 973 ts_tablet_manager.cc:1397] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260924 15:18:07.771773 973 raft_consensus.cc:348] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } }
I20260924 15:18:07.771878 973 raft_consensus.cc:374] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:07.771908 973 raft_consensus.cc:729] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc999b61b7a4db1802dc9279840129c, State: Initialized, Role: FOLLOWER
I20260924 15:18:07.771952 973 consensus_queue.cc:260] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } }
I20260924 15:18:07.772043 973 ts_tablet_manager.cc:1428] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 15:18:07.772922 994 ts_tablet_manager.cc:1918] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:18:07.773016 994 log.cc:1196] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/8692755dd3c441fdaecae7b98f2d9524
I20260924 15:18:07.773334 994 ts_tablet_manager.cc:1939] T 8692755dd3c441fdaecae7b98f2d9524 P bef4c4e4b06e4113b313c8df1aac9ccb: Deleting consensus metadata
I20260924 15:18:07.773718 439 catalog_manager.cc:4841] TS bef4c4e4b06e4113b313c8df1aac9ccb (127.31.12.67:41173): tablet 8692755dd3c441fdaecae7b98f2d9524 (table test-table [id=bd41f584747745588f6e65d261a75b8a]) successfully deleted
I20260924 15:18:07.777824 972 tablet_bootstrap.cc:492] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e: No bootstrap required, opened a new log
I20260924 15:18:07.777872 972 ts_tablet_manager.cc:1397] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260924 15:18:07.778047 972 raft_consensus.cc:348] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } }
I20260924 15:18:07.778108 972 raft_consensus.cc:374] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:07.778121 972 raft_consensus.cc:729] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f7e7edf1af84b6bb40550d953467e9e, State: Initialized, Role: FOLLOWER
I20260924 15:18:07.778174 972 consensus_queue.cc:260] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } }
I20260924 15:18:07.778290 972 ts_tablet_manager.cc:1428] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:07.805691 979 raft_consensus.cc:482] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:18:07.805802 979 raft_consensus.cc:504] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } }
I20260924 15:18:07.806167 979 leader_election.cc:288] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc999b61b7a4db1802dc9279840129c (127.31.12.65:37673), 0f7e7edf1af84b6bb40550d953467e9e (127.31.12.66:33859)
I20260924 15:18:07.809631 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.806460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:40937 (local address 127.31.12.66:33859)
0924 15:18:07.806516 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:07.806519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:07.806586 (+ 67us) server_negotiation.cc:407] Connection header received
0924 15:18:07.806666 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:07.806668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:07.806677 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:07.806726 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:07.806964 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.807673 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.808885 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.809238 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.809320 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.809329 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:07.809513 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 15:18:07.809561 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 15:18:07.809676 839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.806335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:37673 (local address 127.31.12.67:39589)
0924 15:18:07.806418 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:07.806428 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:07.806474 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:07.806711 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:07.806711 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:07.806713 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:07.806783 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.806785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.807453 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.807454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.808593 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.808595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.809372 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.809373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.809446 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.809449 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:18:07.809522 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 15:18:07.809550 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 15:18:07.809963 969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.806557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:39589 (local address 127.31.12.65:37673)
0924 15:18:07.806600 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:07.806602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:07.806609 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:18:07.806638 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:07.806639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:07.806650 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:07.806704 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:07.806856 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.807354 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.808702 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.809286 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.809358 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.809367 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:07.809850 (+ 483us) server_negotiation.cc:299] Negotiation successful
0924 15:18:07.809891 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:18:07.809990 999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.806335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:33859 (local address 127.31.12.67:40937)
0924 15:18:07.806521 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:07.806532 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:07.806599 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:07.806766 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:07.806767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:07.806769 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:07.806880 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.806881 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.807761 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.807762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.808806 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.808808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.809319 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.809320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.809394 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.809398 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:18:07.809501 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 15:18:07.809545 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":13,"thread_start_us":66,"threads_started":1}
I20260924 15:18:07.810202 759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e23f5081c4f44ebbb9c4a60a8d301af" candidate_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f7e7edf1af84b6bb40550d953467e9e" is_pre_election: true
I20260924 15:18:07.810192 617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e23f5081c4f44ebbb9c4a60a8d301af" candidate_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc999b61b7a4db1802dc9279840129c" is_pre_election: true
I20260924 15:18:07.810292 759 raft_consensus.cc:2454] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bef4c4e4b06e4113b313c8df1aac9ccb in term 0.
I20260924 15:18:07.810309 617 raft_consensus.cc:2454] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bef4c4e4b06e4113b313c8df1aac9ccb in term 0.
I20260924 15:18:07.810461 831 leader_election.cc:302] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f7e7edf1af84b6bb40550d953467e9e, bef4c4e4b06e4113b313c8df1aac9ccb; no voters:
I20260924 15:18:07.810602 979 raft_consensus.cc:2781] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:18:07.810633 979 raft_consensus.cc:482] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:18:07.810653 979 raft_consensus.cc:3037] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:07.811507 979 raft_consensus.cc:504] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } }
I20260924 15:18:07.811640 979 leader_election.cc:288] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [CANDIDATE]: Term 1 election: Requested vote from peers fdc999b61b7a4db1802dc9279840129c (127.31.12.65:37673), 0f7e7edf1af84b6bb40550d953467e9e (127.31.12.66:33859)
I20260924 15:18:07.811801 759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e23f5081c4f44ebbb9c4a60a8d301af" candidate_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f7e7edf1af84b6bb40550d953467e9e"
I20260924 15:18:07.811877 759 raft_consensus.cc:3037] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:07.812493 759 raft_consensus.cc:2454] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bef4c4e4b06e4113b313c8df1aac9ccb in term 1.
I20260924 15:18:07.812630 617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e23f5081c4f44ebbb9c4a60a8d301af" candidate_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc999b61b7a4db1802dc9279840129c"
I20260924 15:18:07.812670 831 leader_election.cc:302] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f7e7edf1af84b6bb40550d953467e9e, bef4c4e4b06e4113b313c8df1aac9ccb; no voters:
I20260924 15:18:07.812711 617 raft_consensus.cc:3037] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:07.812772 979 raft_consensus.cc:2781] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:07.812906 979 raft_consensus.cc:686] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [term 1 LEADER]: Becoming Leader. State: Replica: bef4c4e4b06e4113b313c8df1aac9ccb, State: Running, Role: LEADER
I20260924 15:18:07.812978 979 consensus_queue.cc:237] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } }
I20260924 15:18:07.813655 617 raft_consensus.cc:2454] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bef4c4e4b06e4113b313c8df1aac9ccb in term 1.
I20260924 15:18:07.813941 455 catalog_manager.cc:5495] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb reported cstate change: term changed from 0 to 1, leader changed from <none> to bef4c4e4b06e4113b313c8df1aac9ccb (127.31.12.67). New cstate: current_term: 1 leader_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } health_report { overall_health: HEALTHY } } }
W20260924 15:18:07.842031 665 tablet.cc:2369] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 15:18:07 dist-test-slave-zwbb krb5kdc[410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263087, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:07.854058 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.846215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:34383 (local address 127.0.0.1:51426)
0924 15:18:07.846417 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:07.846433 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:07.846493 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:07.846650 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:07.846651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:07.846935 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:07.846981 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.846983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.847930 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.847931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.848573 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:07.848575 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.849052 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.849052 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:07.849097 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.849745 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:07.849749 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:07.851258 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:07.852903 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:07.852909 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:07.852911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:07.853168 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:07.853376 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:07.853376 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:07.853376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:07.853481 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:07.853768 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:07.853774 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:07.853820 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 15:18:07.853891 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 15:18:07.853986 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":98,"threads_started":1}
I20260924 15:18:07.854156 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:07.846271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51426 (local address 127.31.12.126:34383)
0924 15:18:07.846353 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:07.846355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:07.846506 (+ 151us) server_negotiation.cc:407] Connection header received
0924 15:18:07.846545 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:07.846546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:07.846560 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:07.846643 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:07.847066 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.847836 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.848671 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:07.848965 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:07.849035 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:07.851344 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:07.851345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:07.851346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:07.851353 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:07.852823 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:07.853243 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:07.853244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:07.853245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:07.853307 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:07.853535 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:07.853535 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:07.853536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:07.853667 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:07.853738 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:07.853933 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 15:18:07.854046 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":50}
I20260924 15:18:07.856535 455 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:51426:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 15:18:07.856765 455 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:18:07.858237 596 tablet_service.cc:1467] Processing CreateTablet for tablet ca323c46b99240abb28dc214fc91c6a7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2002cdcae06644b087bddf1ed53e7226]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:18:07.858266 738 tablet_service.cc:1467] Processing CreateTablet for tablet ca323c46b99240abb28dc214fc91c6a7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2002cdcae06644b087bddf1ed53e7226]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:18:07.858415 596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca323c46b99240abb28dc214fc91c6a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:07.858424 738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca323c46b99240abb28dc214fc91c6a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:07.859712 972 tablet_bootstrap.cc:492] T ca323c46b99240abb28dc214fc91c6a7 P 0f7e7edf1af84b6bb40550d953467e9e: Bootstrap starting.
I20260924 15:18:07.859982 973 tablet_bootstrap.cc:492] T ca323c46b99240abb28dc214fc91c6a7 P fdc999b61b7a4db1802dc9279840129c: Bootstrap starting.
I20260924 15:18:07.860054 880 tablet_service.cc:1467] Processing CreateTablet for tablet ca323c46b99240abb28dc214fc91c6a7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2002cdcae06644b087bddf1ed53e7226]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:18:07.860193 880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca323c46b99240abb28dc214fc91c6a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:07.860267 972 tablet_bootstrap.cc:654] T ca323c46b99240abb28dc214fc91c6a7 P 0f7e7edf1af84b6bb40550d953467e9e: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:07.860423 973 tablet_bootstrap.cc:654] T ca323c46b99240abb28dc214fc91c6a7 P fdc999b61b7a4db1802dc9279840129c: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:07.861006 972 tablet_bootstrap.cc:492] T ca323c46b99240abb28dc214fc91c6a7 P 0f7e7edf1af84b6bb40550d953467e9e: No bootstrap required, opened a new log
I20260924 15:18:07.861070 972 ts_tablet_manager.cc:1397] T ca323c46b99240abb28dc214fc91c6a7 P 0f7e7edf1af84b6bb40550d953467e9e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:07.861235 972 raft_consensus.cc:348] T ca323c46b99240abb28dc214fc91c6a7 P 0f7e7edf1af84b6bb40550d953467e9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } }
I20260924 15:18:07.861320 972 raft_consensus.cc:374] T ca323c46b99240abb28dc214fc91c6a7 P 0f7e7edf1af84b6bb40550d953467e9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:07.861348 972 raft_consensus.cc:729] T ca323c46b99240abb28dc214fc91c6a7 P 0f7e7edf1af84b6bb40550d953467e9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f7e7edf1af84b6bb40550d953467e9e, State: Initialized, Role: FOLLOWER
I20260924 15:18:07.861408 972 consensus_queue.cc:260] T ca323c46b99240abb28dc214fc91c6a7 P 0f7e7edf1af84b6bb40550d953467e9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } }
I20260924 15:18:07.861547 972 ts_tablet_manager.cc:1428] T ca323c46b99240abb28dc214fc91c6a7 P 0f7e7edf1af84b6bb40550d953467e9e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 15:18:07.861666 971 tablet_bootstrap.cc:492] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb: Bootstrap starting.
I20260924 15:18:07.861689 977 tablet_replica.cc:439] T ca323c46b99240abb28dc214fc91c6a7 P 0f7e7edf1af84b6bb40550d953467e9e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } } }
I20260924 15:18:07.861760 977 tablet_replica.cc:442] T ca323c46b99240abb28dc214fc91c6a7 P 0f7e7edf1af84b6bb40550d953467e9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:18:07.862124 971 tablet_bootstrap.cc:654] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:07.862362 973 tablet_bootstrap.cc:492] T ca323c46b99240abb28dc214fc91c6a7 P fdc999b61b7a4db1802dc9279840129c: No bootstrap required, opened a new log
I20260924 15:18:07.862427 973 ts_tablet_manager.cc:1397] T ca323c46b99240abb28dc214fc91c6a7 P fdc999b61b7a4db1802dc9279840129c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 15:18:07.862581 973 raft_consensus.cc:348] T ca323c46b99240abb28dc214fc91c6a7 P fdc999b61b7a4db1802dc9279840129c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } }
I20260924 15:18:07.862673 973 raft_consensus.cc:374] T ca323c46b99240abb28dc214fc91c6a7 P fdc999b61b7a4db1802dc9279840129c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:07.862691 973 raft_consensus.cc:729] T ca323c46b99240abb28dc214fc91c6a7 P fdc999b61b7a4db1802dc9279840129c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdc999b61b7a4db1802dc9279840129c, State: Initialized, Role: FOLLOWER
I20260924 15:18:07.862727 973 consensus_queue.cc:260] T ca323c46b99240abb28dc214fc91c6a7 P fdc999b61b7a4db1802dc9279840129c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } }
I20260924 15:18:07.862829 973 ts_tablet_manager.cc:1428] T ca323c46b99240abb28dc214fc91c6a7 P fdc999b61b7a4db1802dc9279840129c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:07.862835 971 tablet_bootstrap.cc:492] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb: No bootstrap required, opened a new log
I20260924 15:18:07.862893 971 ts_tablet_manager.cc:1397] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 15:18:07.862865 978 tablet_replica.cc:439] T ca323c46b99240abb28dc214fc91c6a7 P fdc999b61b7a4db1802dc9279840129c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } } }
I20260924 15:18:07.862957 978 tablet_replica.cc:442] T ca323c46b99240abb28dc214fc91c6a7 P fdc999b61b7a4db1802dc9279840129c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:18:07.863041 971 raft_consensus.cc:348] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } }
I20260924 15:18:07.863104 971 raft_consensus.cc:374] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:07.863126 971 raft_consensus.cc:729] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bef4c4e4b06e4113b313c8df1aac9ccb, State: Initialized, Role: FOLLOWER
I20260924 15:18:07.863169 971 consensus_queue.cc:260] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } }
I20260924 15:18:07.863266 971 ts_tablet_manager.cc:1428] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 15:18:07.863376 979 tablet_replica.cc:439] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } } }
I20260924 15:18:07.863445 979 tablet_replica.cc:442] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 15:18:07.917029 949 tablet.cc:2369] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:18:08.010843 807 tablet.cc:2369] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:18:08.052843 979 raft_consensus.cc:482] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:18:08.052940 979 raft_consensus.cc:504] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } }
I20260924 15:18:08.053145 979 leader_election.cc:288] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fdc999b61b7a4db1802dc9279840129c (127.31.12.65:37673), 0f7e7edf1af84b6bb40550d953467e9e (127.31.12.66:33859)
I20260924 15:18:08.053366 617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca323c46b99240abb28dc214fc91c6a7" candidate_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc999b61b7a4db1802dc9279840129c" is_pre_election: true
I20260924 15:18:08.053395 759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca323c46b99240abb28dc214fc91c6a7" candidate_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f7e7edf1af84b6bb40550d953467e9e" is_pre_election: true
I20260924 15:18:08.053508 617 raft_consensus.cc:2454] T ca323c46b99240abb28dc214fc91c6a7 P fdc999b61b7a4db1802dc9279840129c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bef4c4e4b06e4113b313c8df1aac9ccb in term 0.
I20260924 15:18:08.053562 759 raft_consensus.cc:2454] T ca323c46b99240abb28dc214fc91c6a7 P 0f7e7edf1af84b6bb40550d953467e9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bef4c4e4b06e4113b313c8df1aac9ccb in term 0.
I20260924 15:18:08.053740 832 leader_election.cc:302] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bef4c4e4b06e4113b313c8df1aac9ccb, fdc999b61b7a4db1802dc9279840129c; no voters:
I20260924 15:18:08.053882 979 raft_consensus.cc:2781] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:18:08.053943 979 raft_consensus.cc:482] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:18:08.053960 979 raft_consensus.cc:3037] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:08.054677 979 raft_consensus.cc:504] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } }
I20260924 15:18:08.054806 979 leader_election.cc:288] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb [CANDIDATE]: Term 1 election: Requested vote from peers fdc999b61b7a4db1802dc9279840129c (127.31.12.65:37673), 0f7e7edf1af84b6bb40550d953467e9e (127.31.12.66:33859)
I20260924 15:18:08.055006 617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca323c46b99240abb28dc214fc91c6a7" candidate_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdc999b61b7a4db1802dc9279840129c"
I20260924 15:18:08.055012 759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca323c46b99240abb28dc214fc91c6a7" candidate_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f7e7edf1af84b6bb40550d953467e9e"
I20260924 15:18:08.055085 759 raft_consensus.cc:3037] T ca323c46b99240abb28dc214fc91c6a7 P 0f7e7edf1af84b6bb40550d953467e9e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:08.055084 617 raft_consensus.cc:3037] T ca323c46b99240abb28dc214fc91c6a7 P fdc999b61b7a4db1802dc9279840129c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:08.055819 617 raft_consensus.cc:2454] T ca323c46b99240abb28dc214fc91c6a7 P fdc999b61b7a4db1802dc9279840129c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bef4c4e4b06e4113b313c8df1aac9ccb in term 1.
I20260924 15:18:08.055819 759 raft_consensus.cc:2454] T ca323c46b99240abb28dc214fc91c6a7 P 0f7e7edf1af84b6bb40550d953467e9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bef4c4e4b06e4113b313c8df1aac9ccb in term 1.
I20260924 15:18:08.056005 832 leader_election.cc:302] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bef4c4e4b06e4113b313c8df1aac9ccb, fdc999b61b7a4db1802dc9279840129c; no voters:
I20260924 15:18:08.056126 979 raft_consensus.cc:2781] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:08.056175 979 raft_consensus.cc:686] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb [term 1 LEADER]: Becoming Leader. State: Replica: bef4c4e4b06e4113b313c8df1aac9ccb, State: Running, Role: LEADER
I20260924 15:18:08.056245 979 consensus_queue.cc:237] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } }
I20260924 15:18:08.056579 1000 tablet_replica.cc:439] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb: TxnStatusTablet state changed. Reason: New leader bef4c4e4b06e4113b313c8df1aac9ccb. Latest consensus state: current_term: 1 leader_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } } }
I20260924 15:18:08.056670 1000 tablet_replica.cc:442] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:18:08.056876 1012 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 15:18:08.057025 1012 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 15:18:08.056936 455 catalog_manager.cc:5495] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb reported cstate change: term changed from 0 to 1, leader changed from <none> to bef4c4e4b06e4113b313c8df1aac9ccb (127.31.12.67). New cstate: current_term: 1 leader_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } health_report { overall_health: HEALTHY } } }
I20260924 15:18:08.090901 970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:08.088399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47536 (local address 127.31.12.67:41173)
0924 15:18:08.088449 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:08.088451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:08.088461 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:18:08.088543 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:08.088544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:08.088562 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:08.088639 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:18:08.088766 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:08.089250 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:08.090079 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:08.090286 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:08.090361 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:08.090575 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:18:08.090675 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:18:08.090749 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:08.090804 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 15:18:08.090826 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 15:18:08.090919 1007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:08.088312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:41173 (local address 127.0.0.1:47536)
0924 15:18:08.088384 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:08.088396 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:08.088458 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:08.088643 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:08.088644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:08.088645 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:18:08.088698 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:08.088700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:08.089333 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:08.089334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:08.089989 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:08.089992 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:08.090358 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:08.090359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:08.090460 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:08.090474 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:18:08.090747 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:18:08.090757 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 15:18:08.090802 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 15:18:08.090852 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 15:18:08.092142 759 raft_consensus.cc:1261] T ca323c46b99240abb28dc214fc91c6a7 P 0f7e7edf1af84b6bb40550d953467e9e [term 1 FOLLOWER]: Refusing update from remote peer bef4c4e4b06e4113b313c8df1aac9ccb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:08.092163 617 raft_consensus.cc:1261] T ca323c46b99240abb28dc214fc91c6a7 P fdc999b61b7a4db1802dc9279840129c [term 1 FOLLOWER]: Refusing update from remote peer bef4c4e4b06e4113b313c8df1aac9ccb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:08.092435 1000 consensus_queue.cc:1035] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:08.092551 979 consensus_queue.cc:1035] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:08.093081 977 tablet_replica.cc:439] T ca323c46b99240abb28dc214fc91c6a7 P 0f7e7edf1af84b6bb40550d953467e9e: TxnStatusTablet state changed. Reason: New leader bef4c4e4b06e4113b313c8df1aac9ccb. Latest consensus state: current_term: 1 leader_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } } }
I20260924 15:18:08.093165 977 tablet_replica.cc:442] T ca323c46b99240abb28dc214fc91c6a7 P 0f7e7edf1af84b6bb40550d953467e9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:18:08.093235 978 tablet_replica.cc:439] T ca323c46b99240abb28dc214fc91c6a7 P fdc999b61b7a4db1802dc9279840129c: TxnStatusTablet state changed. Reason: New leader bef4c4e4b06e4113b313c8df1aac9ccb. Latest consensus state: current_term: 1 leader_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } } }
I20260924 15:18:08.093334 978 tablet_replica.cc:442] T ca323c46b99240abb28dc214fc91c6a7 P fdc999b61b7a4db1802dc9279840129c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:18:08.093761 979 tablet_replica.cc:439] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } } }
I20260924 15:18:08.093851 979 tablet_replica.cc:442] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:18:08.094296 984 mvcc.cc:205] Tried to move back new op lower bound from 7332917608823144448 to 7332917608678813696. Current Snapshot: MvccSnapshot[applied={T|T < 7332917608823144448}]
I20260924 15:18:08.094401 978 tablet_replica.cc:439] T ca323c46b99240abb28dc214fc91c6a7 P fdc999b61b7a4db1802dc9279840129c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } } }
I20260924 15:18:08.094475 978 tablet_replica.cc:442] T ca323c46b99240abb28dc214fc91c6a7 P fdc999b61b7a4db1802dc9279840129c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:18:08.095369 1000 tablet_replica.cc:439] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } } }
I20260924 15:18:08.095464 1000 tablet_replica.cc:442] T ca323c46b99240abb28dc214fc91c6a7 P bef4c4e4b06e4113b313c8df1aac9ccb: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:18:08.095929 977 tablet_replica.cc:439] T ca323c46b99240abb28dc214fc91c6a7 P 0f7e7edf1af84b6bb40550d953467e9e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 } } peers { permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 } } peers { permanent_uuid: "bef4c4e4b06e4113b313c8df1aac9ccb" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41173 } } }
I20260924 15:18:08.096022 977 tablet_replica.cc:442] T ca323c46b99240abb28dc214fc91c6a7 P 0f7e7edf1af84b6bb40550d953467e9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:18:08.098577 970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:08.096179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47552 (local address 127.31.12.67:41173)
0924 15:18:08.096211 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:08.096213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:08.096219 (+ 6us) server_negotiation.cc:407] Connection header received
0924 15:18:08.096235 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:08.096236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:08.096243 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:08.096291 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:18:08.096429 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:08.097150 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:08.097903 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:08.098199 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:08.098268 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:08.098373 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:18:08.098403 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:18:08.098457 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:08.098495 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 15:18:08.098513 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:18:08.103125 970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:08.100835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:54105 (local address 127.31.12.67:41173)
0924 15:18:08.100870 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:08.100872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:08.100905 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:18:08.100969 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:08.100970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:08.100978 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:08.101033 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:18:08.101149 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:08.101650 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:08.102420 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:08.102618 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:08.102705 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:08.102880 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:18:08.102912 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:18:08.102968 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:08.103025 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 15:18:08.103052 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":21}
I20260924 15:18:08.103180 954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:08.100789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:41173 (local address 127.31.12.67:54105)
0924 15:18:08.100866 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:08.100876 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:08.100918 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:08.101032 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:08.101032 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:08.101033 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:18:08.101089 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:08.101090 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:08.101732 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:08.101733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:08.102326 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:08.102328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:08.102720 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:08.102721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:08.102783 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:08.102795 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:18:08.102983 (+ 188us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:18:08.102984 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 15:18:08.103025 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 15:18:08.103076 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":11}
I20260924 15:18:08.105147 759 raft_consensus.cc:1261] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e [term 1 FOLLOWER]: Refusing update from remote peer bef4c4e4b06e4113b313c8df1aac9ccb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:08.105146 617 raft_consensus.cc:1261] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c [term 1 FOLLOWER]: Refusing update from remote peer bef4c4e4b06e4113b313c8df1aac9ccb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:08.105350 1017 consensus_queue.cc:1035] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f7e7edf1af84b6bb40550d953467e9e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:08.105484 1000 consensus_queue.cc:1035] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdc999b61b7a4db1802dc9279840129c" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 37673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:08.115934 454 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:18:08.116034 454 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:18:08.116637 454 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57fde51b60654080b094758aebd8005b: Sending DeleteTablet for 3 replicas of tablet 8e23f5081c4f44ebbb9c4a60a8d301af
I20260924 15:18:08.116899 738 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e23f5081c4f44ebbb9c4a60a8d301af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:08 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:53248
I20260924 15:18:08.116914 596 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e23f5081c4f44ebbb9c4a60a8d301af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:08 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:49658
I20260924 15:18:08.117034 995 tablet_replica.cc:330] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e: stopping tablet replica
I20260924 15:18:08.117091 995 raft_consensus.cc:2229] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:18:08.117130 995 raft_consensus.cc:2258] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:08.117202 993 tablet_replica.cc:330] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c: stopping tablet replica
I20260924 15:18:08.117260 993 raft_consensus.cc:2229] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:18:08.117300 993 raft_consensus.cc:2258] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:08.117578 879 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e23f5081c4f44ebbb9c4a60a8d301af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:08 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:47524
I20260924 15:18:08.117664 993 ts_tablet_manager.cc:1905] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:08.117681 994 tablet_replica.cc:330] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb: stopping tablet replica
I20260924 15:18:08.117731 994 raft_consensus.cc:2229] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:18:08.117805 994 raft_consensus.cc:2258] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:08.118060 994 ts_tablet_manager.cc:1905] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:08.119362 993 ts_tablet_manager.cc:1918] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:18:08.119448 993 log.cc:1196] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-0/wal/wals/8e23f5081c4f44ebbb9c4a60a8d301af
I20260924 15:18:08.119556 994 ts_tablet_manager.cc:1918] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:18:08.119618 994 log.cc:1196] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/8e23f5081c4f44ebbb9c4a60a8d301af
I20260924 15:18:08.119707 993 ts_tablet_manager.cc:1939] T 8e23f5081c4f44ebbb9c4a60a8d301af P fdc999b61b7a4db1802dc9279840129c: Deleting consensus metadata
I20260924 15:18:08.119837 994 ts_tablet_manager.cc:1939] T 8e23f5081c4f44ebbb9c4a60a8d301af P bef4c4e4b06e4113b313c8df1aac9ccb: Deleting consensus metadata
I20260924 15:18:08.120010 440 catalog_manager.cc:4841] TS fdc999b61b7a4db1802dc9279840129c (127.31.12.65:37673): tablet 8e23f5081c4f44ebbb9c4a60a8d301af (table test-table [id=5bbea4071bfd4037bc3548a5c89c4612]) successfully deleted
I20260924 15:18:08.120108 439 catalog_manager.cc:4841] TS bef4c4e4b06e4113b313c8df1aac9ccb (127.31.12.67:41173): tablet 8e23f5081c4f44ebbb9c4a60a8d301af (table test-table [id=5bbea4071bfd4037bc3548a5c89c4612]) successfully deleted
I20260924 15:18:08.120205 995 ts_tablet_manager.cc:1905] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:08.121686 995 ts_tablet_manager.cc:1918] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:18:08.121747 995 log.cc:1196] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/8e23f5081c4f44ebbb9c4a60a8d301af
I20260924 15:18:08.121949 995 ts_tablet_manager.cc:1939] T 8e23f5081c4f44ebbb9c4a60a8d301af P 0f7e7edf1af84b6bb40550d953467e9e: Deleting consensus metadata
I20260924 15:18:08.122385 439 catalog_manager.cc:4841] TS 0f7e7edf1af84b6bb40550d953467e9e (127.31.12.66:33859): tablet 8e23f5081c4f44ebbb9c4a60a8d301af (table test-table [id=5bbea4071bfd4037bc3548a5c89c4612]) successfully deleted
Sep 24 15:18:08 dist-test-slave-zwbb krb5kdc[410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263087, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.12.65@KRBTEST.COM
I20260924 15:18:08.130563 969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:08.117555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49678 (local address 127.31.12.65:37673)
0924 15:18:08.117593 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:08.117595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:08.121703 (+ 4108us) server_negotiation.cc:407] Connection header received
0924 15:18:08.121799 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:08.121800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:08.121812 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:08.121842 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:08.122592 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:08.123109 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:08.124198 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:08.124439 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:08.124493 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:08.127627 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:08.127635 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:08.127635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:08.127643 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:08.129405 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:08.129784 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:08.129787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:08.129788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:08.129853 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:08.130037 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:08.130038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:08.130039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:08.130198 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:08.130335 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:08.130381 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 15:18:08.130472 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 15:18:08.130832 645 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49678
I20260924 15:18:08.138290 526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:08.131777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51434 (local address 127.31.12.126:34383)
0924 15:18:08.131825 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:08.131827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:08.131913 (+ 86us) server_negotiation.cc:407] Connection header received
0924 15:18:08.131973 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:08.131974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:08.131983 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:08.132027 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:08.132651 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:08.133120 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:08.134106 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:08.134286 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:08.134350 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:08.135635 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:08.135636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:08.135637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:08.135643 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:08.137119 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:08.137575 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:08.137576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:08.137577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:08.137630 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:08.137840 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:08.137840 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:08.137840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:08.137945 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:08.138020 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:08.138162 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 15:18:08.138229 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 15:18:08.138774 454 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51434
I20260924 15:18:08.139366 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 530
I20260924 15:18:08.148847 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 673
I20260924 15:18:08.157054 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 815
I20260924 15:18:08.162436 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 423
2026-09-24T15:18:08Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2292 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:18:08 dist-test-slave-zwbb krb5kdc[1029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:18:08 dist-test-slave-zwbb krb5kdc[1029](info): set up 2 sockets
Sep 24 15:18:08 dist-test-slave-zwbb krb5kdc[1029](info): commencing operation
krb5kdc: starting...
W20260924 15:18:09.207050 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:18:09 dist-test-slave-zwbb krb5kdc[1029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:18:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:18:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:18:09.324209 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:39359
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:39359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:09.402905 1042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:09.403097 1042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:09.403118 1042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:09.404269 1042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:18:09.404310 1042 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 15:18:09.404325 1042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:09.404337 1042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:09.404377 1042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:18:09.404408 1042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:18:09.405860 1042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:39359
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:39359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:09.406108 1042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:09.406352 1042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:09.411159 1047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:09.411187 1050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:09.411228 1042 server_base.cc:1034] running on GCE node
W20260924 15:18:09.411187 1048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:09.411559 1042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:09.411796 1042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:09.412979 1042 hybrid_clock.cc:648] HybridClock initialized: now 1790263089412962 us; error 43 us; skew 500 ppm
Sep 24 15:18:09 dist-test-slave-zwbb krb5kdc[1029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263089, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:09.415428 1042 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:18:09.415795 1042 webserver.cc:460] Webserver started at http://127.31.12.126:38695/ using document root <none> and password file <none>
I20260924 15:18:09.415967 1042 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:09.416005 1042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:09.416090 1042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:09.417039 1042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "39eb135056df46afa99ddce9dd14a788"
format_stamp: "Formatted at 2026-09-24 15:18:09 on dist-test-slave-zwbb"
server_key: "34006d6d69a96d50c2287b288a6b6ad3"
server_key_iv: "bd55484ec540bf8eca1e4ab23d22b30c"
server_key_version: "encryptionkey@0"
I20260924 15:18:09.417335 1042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "39eb135056df46afa99ddce9dd14a788"
format_stamp: "Formatted at 2026-09-24 15:18:09 on dist-test-slave-zwbb"
server_key: "34006d6d69a96d50c2287b288a6b6ad3"
server_key_iv: "bd55484ec540bf8eca1e4ab23d22b30c"
server_key_version: "encryptionkey@0"
I20260924 15:18:09.418684 1042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:18:09.419394 1057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:09.419571 1042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:18:09.419654 1042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "39eb135056df46afa99ddce9dd14a788"
format_stamp: "Formatted at 2026-09-24 15:18:09 on dist-test-slave-zwbb"
server_key: "34006d6d69a96d50c2287b288a6b6ad3"
server_key_iv: "bd55484ec540bf8eca1e4ab23d22b30c"
server_key_version: "encryptionkey@0"
I20260924 15:18:09.419709 1042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:09.428458 1042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:09.430227 1042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:09.430377 1042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:09.435015 1042 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:39359
I20260924 15:18:09.435113 1121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:39359 every 8 connection(s)
I20260924 15:18:09.435410 1042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:18:09.436058 1122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:09.438223 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 1042
I20260924 15:18:09.438325 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:18:09.438355 1122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788: Bootstrap starting.
I20260924 15:18:09.438510 31793 external_mini_cluster.cc:1428] Setting key 1e2a47474383477ae8025102a04140f9
I20260924 15:18:09.439038 1122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:09.439364 1122 log.cc:824] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:09.440105 1122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788: No bootstrap required, opened a new log
I20260924 15:18:09.442456 1122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39eb135056df46afa99ddce9dd14a788" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 39359 } }
I20260924 15:18:09.442584 1122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:09.442605 1122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39eb135056df46afa99ddce9dd14a788, State: Initialized, Role: FOLLOWER
I20260924 15:18:09.442732 1122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39eb135056df46afa99ddce9dd14a788" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 39359 } }
I20260924 15:18:09.442800 1122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:18:09.442822 1122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:18:09.442901 1122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:09.443573 1122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39eb135056df46afa99ddce9dd14a788" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 39359 } }
I20260924 15:18:09.443679 1122 leader_election.cc:302] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39eb135056df46afa99ddce9dd14a788; no voters:
I20260924 15:18:09.443872 1122 leader_election.cc:288] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:18:09.443943 1127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:09.444120 1127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788 [term 1 LEADER]: Becoming Leader. State: Replica: 39eb135056df46afa99ddce9dd14a788, State: Running, Role: LEADER
I20260924 15:18:09.444129 1122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:18:09.444906 1127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39eb135056df46afa99ddce9dd14a788" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 39359 } }
I20260924 15:18:09.445605 1129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39eb135056df46afa99ddce9dd14a788. Latest consensus state: current_term: 1 leader_uuid: "39eb135056df46afa99ddce9dd14a788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39eb135056df46afa99ddce9dd14a788" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 39359 } } }
I20260924 15:18:09.446058 1129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788 [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:09.445761 1128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39eb135056df46afa99ddce9dd14a788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39eb135056df46afa99ddce9dd14a788" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 39359 } } }
I20260924 15:18:09.446208 1128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788 [sys.catalog]: This master's current role is: LEADER
W20260924 15:18:09.446316 1141 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:18:09.446369 1141 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 24 15:18:09 dist-test-slave-zwbb krb5kdc[1029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:09.446467 1145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:18:09.446630 1145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:18:09.448132 1145 catalog_manager.cc:1347] Generated new cluster ID: e3b61eaba09d4dd4bb7c33f6157a6679
I20260924 15:18:09.448203 1145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:18:09.450389 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.438993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34756 (local address 127.31.12.126:39359)
0924 15:18:09.439218 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:09.439229 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:09.439262 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:18:09.439856 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:09.439865 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:09.439927 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:09.440073 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:09.440728 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.442209 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.443129 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.443763 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.444198 (+ 435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.446894 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:09.446898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:09.446899 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:09.446909 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:09.448810 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:09.449196 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:09.449199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:09.449199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:09.449302 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:09.449507 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:09.449508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:09.449508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:09.449801 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:09.449957 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:09.450063 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 15:18:09.450194 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":71,"threads_started":1}
I20260924 15:18:09.454866 1145 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:18:09.455605 1145 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:18:09.459353 1145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788: Generated new TSK 0
I20260924 15:18:09.459506 1145 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
I20260924 15:18:09.486440 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:39359
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:09.567916 1149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:09.568117 1149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:09.568141 1149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:09.569276 1149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:09.569316 1149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:09.569336 1149 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 15:18:09.569350 1149 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:18:09.569366 1149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:18:09.570942 1149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.12.126:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:09.571189 1149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:09.571429 1149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:09.576382 1157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:09.576360 1154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:09.576375 1155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:09.576646 1149 server_base.cc:1034] running on GCE node
I20260924 15:18:09.576871 1149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:09.577135 1149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:09.578289 1149 hybrid_clock.cc:648] HybridClock initialized: now 1790263089578279 us; error 31 us; skew 500 ppm
Sep 24 15:18:09 dist-test-slave-zwbb krb5kdc[1029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263089, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:09.580907 1149 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:18:09.581387 1149 webserver.cc:460] Webserver started at http://127.31.12.65:41879/ using document root <none> and password file <none>
I20260924 15:18:09.581672 1149 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:09.581746 1149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:09.581894 1149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:09.583024 1149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "1a280bc8a849478da774f8bae065653d"
format_stamp: "Formatted at 2026-09-24 15:18:09 on dist-test-slave-zwbb"
server_key: "1810f4c35dd0c83e2f6b53988e9ed871"
server_key_iv: "58fd3537f1a5fa664ce3606dd1776b1c"
server_key_version: "encryptionkey@0"
I20260924 15:18:09.583381 1149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "1a280bc8a849478da774f8bae065653d"
format_stamp: "Formatted at 2026-09-24 15:18:09 on dist-test-slave-zwbb"
server_key: "1810f4c35dd0c83e2f6b53988e9ed871"
server_key_iv: "58fd3537f1a5fa664ce3606dd1776b1c"
server_key_version: "encryptionkey@0"
I20260924 15:18:09.584729 1149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:18:09.585543 1165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:09.585712 1149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:09.585810 1149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "1a280bc8a849478da774f8bae065653d"
format_stamp: "Formatted at 2026-09-24 15:18:09 on dist-test-slave-zwbb"
server_key: "1810f4c35dd0c83e2f6b53988e9ed871"
server_key_iv: "58fd3537f1a5fa664ce3606dd1776b1c"
server_key_version: "encryptionkey@0"
I20260924 15:18:09.585866 1149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:09.593494 1149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:09.595326 1149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:09.595436 1149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:09.595860 1149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:09.595943 1149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:09.595989 1149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:09.596016 1149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:18:09 dist-test-slave-zwbb krb5kdc[1029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263089, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:09.602770 1149 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:42361
I20260924 15:18:09.603122 1149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:18:09.603255 1281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:42361 every 8 connection(s)
I20260924 15:18:09.605762 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.596323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:38269 (local address 127.31.12.126:39359)
0924 15:18:09.596393 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:09.596395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:09.596602 (+ 207us) server_negotiation.cc:407] Connection header received
0924 15:18:09.597052 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:09.597054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:09.597068 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:09.597159 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:09.598025 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.598890 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.600438 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.600807 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.600884 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.603193 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:09.603194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:09.603195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:09.603202 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:09.604584 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:09.604950 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:09.604951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:09.604951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:09.604999 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:09.605205 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:09.605206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:09.605206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:09.605301 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:09.605348 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:09.605634 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 15:18:09.605700 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:18:09.605911 1174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.596290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:39359 (local address 127.31.12.65:38269)
0924 15:18:09.596469 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:09.596507 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:09.596961 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:09.597206 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:09.597209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:09.597630 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:09.597921 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.597927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.599009 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.599010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.600340 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.600344 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.600914 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.600915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.601063 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.601595 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:09.601605 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:09.603099 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:09.604660 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:09.604665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:09.604667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:09.604887 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:09.605065 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:09.605066 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:09.605066 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:09.605154 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:09.605354 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:09.605358 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:09.605521 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 15:18:09.605604 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 15:18:09.605714 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
I20260924 15:18:09.606388 1282 heartbeater.cc:347] Connected to a master server at 127.31.12.126:39359
I20260924 15:18:09.606468 1282 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:09.606642 1282 heartbeater.cc:511] Master 127.31.12.126:39359 requested a full tablet report, sending...
I20260924 15:18:09.607122 1074 ts_manager.cc:194] Registered new tserver with Master: 1a280bc8a849478da774f8bae065653d (127.31.12.65:42361)
I20260924 15:18:09.607892 1074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:38269
I20260924 15:18:09.611131 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 1149
I20260924 15:18:09.611222 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:18:09.611399 31793 external_mini_cluster.cc:1428] Setting key 323adee977fae214054179b2a4b4f25b
I20260924 15:18:09.613611 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.607379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:39359 (local address 127.31.12.65:40651)
0924 15:18:09.607491 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:09.607520 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:09.607560 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:09.608230 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:09.608231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:09.608446 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:09.608505 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.608507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.609190 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.609190 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.609759 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.609760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.610087 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.610088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.610147 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.610558 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:09.610563 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:09.610820 (+ 257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:09.612484 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:09.612485 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:09.612485 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:09.612709 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:09.612905 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:09.612905 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:09.612906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:09.612993 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:09.613203 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:09.613203 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:09.613250 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 15:18:09.613318 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 15:18:09.613426 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":62,"threads_started":1}
I20260924 15:18:09.613771 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.607441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:40651 (local address 127.31.12.126:39359)
0924 15:18:09.608089 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:09.608091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:09.608099 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:18:09.608121 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:09.608122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:09.608132 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:09.608185 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:09.608588 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.609108 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.609824 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.610010 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.610064 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.610887 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:09.610887 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:09.610888 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:09.610892 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:09.612407 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:09.612777 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:09.612778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:09.612778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:09.612828 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:09.613054 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:09.613055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:09.613055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:09.613144 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:09.613191 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:09.613343 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 15:18:09.613399 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628}
WARNING: no policy specified for kudu/127.31.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
I20260924 15:18:09.636775 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:39359
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:09.717638 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:09.717826 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:09.717851 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:09.718971 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:09.719013 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:09.719034 1293 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 15:18:09.719085 1293 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:18:09.719116 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:18:09.720566 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.12.126:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:09.720798 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:09.721024 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:09.725905 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:09.725914 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:09.725955 1293 server_base.cc:1034] running on GCE node
W20260924 15:18:09.726122 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:09.726326 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:09.726542 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:09.727687 1293 hybrid_clock.cc:648] HybridClock initialized: now 1790263089727674 us; error 35 us; skew 500 ppm
Sep 24 15:18:09 dist-test-slave-zwbb krb5kdc[1029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263089, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:09.730003 1293 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:18:09.730404 1293 webserver.cc:460] Webserver started at http://127.31.12.66:36079/ using document root <none> and password file <none>
I20260924 15:18:09.730619 1293 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:09.730685 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:09.730837 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:09.731859 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2"
format_stamp: "Formatted at 2026-09-24 15:18:09 on dist-test-slave-zwbb"
server_key: "ea3acd503d0f8c8ab7e3bbc9e6a8095a"
server_key_iv: "56a17e55dc8027d9585ca5fa117c4826"
server_key_version: "encryptionkey@0"
I20260924 15:18:09.732237 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2"
format_stamp: "Formatted at 2026-09-24 15:18:09 on dist-test-slave-zwbb"
server_key: "ea3acd503d0f8c8ab7e3bbc9e6a8095a"
server_key_iv: "56a17e55dc8027d9585ca5fa117c4826"
server_key_version: "encryptionkey@0"
I20260924 15:18:09.733582 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:18:09.734267 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:09.734421 1293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:09.734536 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2"
format_stamp: "Formatted at 2026-09-24 15:18:09 on dist-test-slave-zwbb"
server_key: "ea3acd503d0f8c8ab7e3bbc9e6a8095a"
server_key_iv: "56a17e55dc8027d9585ca5fa117c4826"
server_key_version: "encryptionkey@0"
I20260924 15:18:09.734613 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:09.742273 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:09.743999 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:09.744146 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:09.744706 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:09.744771 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:09.744800 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:09.744828 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:09.751515 1293 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:32925
I20260924 15:18:09.751579 1424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:32925 every 8 connection(s)
Sep 24 15:18:09 dist-test-slave-zwbb krb5kdc[1029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263089, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:09.751900 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:18:09.754781 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.745186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:36629 (local address 127.31.12.126:39359)
0924 15:18:09.745257 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:09.745259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:09.745460 (+ 201us) server_negotiation.cc:407] Connection header received
0924 15:18:09.745980 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:09.745981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:09.745994 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:09.746037 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:09.746833 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.747670 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.749210 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.749577 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.749654 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.752344 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:09.752346 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:09.752346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:09.752354 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:09.753690 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:09.754047 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:09.754048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:09.754048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:09.754097 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:09.754311 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:09.754312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:09.754312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:09.754410 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:09.754465 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:09.754661 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 15:18:09.754719 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 15:18:09.754988 1317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.745164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:39359 (local address 127.31.12.66:36629)
0924 15:18:09.745324 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:09.745361 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:09.745850 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:09.746118 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:09.746121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:09.746434 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:09.746712 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.746718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.747773 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.747775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.749103 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.749106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.749667 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.749668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.749777 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.750316 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:09.750345 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:09.752246 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:09.753758 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:09.753764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:09.753766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:09.753982 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:09.754152 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:09.754153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:09.754153 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:09.754244 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:09.754465 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:09.754468 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:09.754568 (+ 100us) client_negotiation.cc:769] Sending connection context
0924 15:18:09.754640 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 15:18:09.754749 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":102,"threads_started":1}
I20260924 15:18:09.755457 1425 heartbeater.cc:347] Connected to a master server at 127.31.12.126:39359
I20260924 15:18:09.755534 1425 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:09.755703 1425 heartbeater.cc:511] Master 127.31.12.126:39359 requested a full tablet report, sending...
I20260924 15:18:09.756104 1074 ts_manager.cc:194] Registered new tserver with Master: e4142cfb5fb54ed5aa8ceaf502688ae2 (127.31.12.66:32925)
I20260924 15:18:09.756582 1074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:36629
I20260924 15:18:09.761390 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 1293
I20260924 15:18:09.761523 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:18:09.761713 31793 external_mini_cluster.cc:1428] Setting key c010e77a1725a6a09dc991e3cc822370
I20260924 15:18:09.761715 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.756547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:36081 (local address 127.31.12.126:39359)
0924 15:18:09.756588 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:09.756590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:09.756649 (+ 59us) server_negotiation.cc:407] Connection header received
0924 15:18:09.756749 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:09.756750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:09.756763 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:09.756823 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:09.757115 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.757636 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.758311 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.758512 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.758583 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.759466 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:09.759467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:09.759468 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:09.759474 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:09.760583 (+ 1109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:09.760915 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:09.760916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:09.760917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:09.760980 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:09.761173 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:09.761174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:09.761174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:09.761319 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:09.761397 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:09.761564 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 15:18:09.761632 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:18:09.761723 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.756479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:39359 (local address 127.31.12.66:36081)
0924 15:18:09.756597 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:09.756608 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:09.756664 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:09.756826 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:09.756827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:09.756980 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:09.757027 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.757028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.757722 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.757723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.758221 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.758223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.758581 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.758582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.758655 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.759085 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:09.759090 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:09.759356 (+ 266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:09.760662 (+ 1306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:09.760664 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:09.760664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:09.760830 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:09.761045 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:09.761046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:09.761047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:09.761102 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:09.761389 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:09.761390 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:09.761461 (+ 71us) client_negotiation.cc:769] Sending connection context
0924 15:18:09.761526 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 15:18:09.761628 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for kudu/127.31.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
I20260924 15:18:09.785881 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:39359
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:34117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:09.865500 1435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:09.865682 1435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:09.865705 1435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:09.866811 1435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:09.866852 1435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:09.866871 1435 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 15:18:09.866883 1435 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:18:09.866900 1435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:18:09.868330 1435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.12.126:39359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:09.868562 1435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:09.868786 1435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:09.873777 1442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:09.873847 1441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:09.873953 1435 server_base.cc:1034] running on GCE node
W20260924 15:18:09.873775 1444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:09.874204 1435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:09.874469 1435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:09.875629 1435 hybrid_clock.cc:648] HybridClock initialized: now 1790263089875619 us; error 45 us; skew 500 ppm
Sep 24 15:18:09 dist-test-slave-zwbb krb5kdc[1029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263089, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:09.878196 1435 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:18:09.878671 1435 webserver.cc:460] Webserver started at http://127.31.12.67:42879/ using document root <none> and password file <none>
I20260924 15:18:09.878911 1435 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:09.878984 1435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:09.879129 1435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:09.880157 1435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "fa584d948dc949f5b4f28e1e29f46da7"
format_stamp: "Formatted at 2026-09-24 15:18:09 on dist-test-slave-zwbb"
server_key: "93fcf1d26ae9a49a6406e5088c135c6f"
server_key_iv: "a8c5fd034847337cc4ff77307db516db"
server_key_version: "encryptionkey@0"
I20260924 15:18:09.880510 1435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "fa584d948dc949f5b4f28e1e29f46da7"
format_stamp: "Formatted at 2026-09-24 15:18:09 on dist-test-slave-zwbb"
server_key: "93fcf1d26ae9a49a6406e5088c135c6f"
server_key_iv: "a8c5fd034847337cc4ff77307db516db"
server_key_version: "encryptionkey@0"
I20260924 15:18:09.881847 1435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:18:09.882691 1451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:09.882843 1435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:09.882958 1435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "fa584d948dc949f5b4f28e1e29f46da7"
format_stamp: "Formatted at 2026-09-24 15:18:09 on dist-test-slave-zwbb"
server_key: "93fcf1d26ae9a49a6406e5088c135c6f"
server_key_iv: "a8c5fd034847337cc4ff77307db516db"
server_key_version: "encryptionkey@0"
I20260924 15:18:09.883031 1435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:09.901885 1435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:09.904124 1435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:09.904284 1435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:09.904819 1435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:09.904889 1435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:09.904945 1435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:09.904968 1435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:18:09 dist-test-slave-zwbb krb5kdc[1029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263089, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:09.912168 1435 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:36935
I20260924 15:18:09.912231 1566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:36935 every 8 connection(s)
I20260924 15:18:09.912578 1435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:18:09.914422 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.905217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:55375 (local address 127.31.12.126:39359)
0924 15:18:09.905280 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:09.905282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:09.905555 (+ 273us) server_negotiation.cc:407] Connection header received
0924 15:18:09.906059 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:09.906061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:09.906074 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:09.906161 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:09.906838 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.907431 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.908415 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.908732 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.908803 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.911505 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:09.911506 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:09.911506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:09.911521 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:09.913190 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:09.913631 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:09.913632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:09.913632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:09.913684 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:09.913891 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:09.913891 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:09.913892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:09.914009 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:09.914059 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:09.914295 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 15:18:09.914358 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:18:09.914553 1460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.905219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:39359 (local address 127.31.12.67:55375)
0924 15:18:09.905384 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:09.905423 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:09.905959 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:09.906209 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:09.906212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:09.906432 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:09.906743 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.906749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.907527 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.907528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.908310 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.908312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.908809 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.908810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.908939 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.909407 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:09.909417 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:09.911396 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:09.913270 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:09.913274 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:09.913275 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:09.913562 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:09.913737 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:09.913737 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:09.913738 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:09.913831 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:09.914067 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:09.914071 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:09.914179 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 15:18:09.914253 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 15:18:09.914367 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":98,"threads_started":1}
I20260924 15:18:09.915082 1567 heartbeater.cc:347] Connected to a master server at 127.31.12.126:39359
I20260924 15:18:09.915180 1567 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:09.915347 1567 heartbeater.cc:511] Master 127.31.12.126:39359 requested a full tablet report, sending...
I20260924 15:18:09.915743 1075 ts_manager.cc:194] Registered new tserver with Master: fa584d948dc949f5b4f28e1e29f46da7 (127.31.12.67:36935)
I20260924 15:18:09.916282 1075 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:55375
I20260924 15:18:09.920449 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 1435
I20260924 15:18:09.920570 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:18:09.920753 31793 external_mini_cluster.cc:1428] Setting key b9d6dbf840c38eb04e2ccf22a6397645
I20260924 15:18:09.921060 1574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.916051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:39359 (local address 127.31.12.67:55175)
0924 15:18:09.916212 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:09.916222 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:09.916300 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:09.916501 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:09.916503 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:09.916705 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:09.916762 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.916764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.917363 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.917364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.917935 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.917937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.918228 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.918228 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.918282 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.918592 (+ 310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:09.918595 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:09.918849 (+ 254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:09.920038 (+ 1189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:09.920039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:09.920039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:09.920194 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:09.920413 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:09.920414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:09.920414 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:09.920480 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:09.920743 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:09.920743 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:09.920785 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 15:18:09.920854 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 15:18:09.920960 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":115,"threads_started":1}
I20260924 15:18:09.921092 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.916119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:55175 (local address 127.31.12.126:39359)
0924 15:18:09.916160 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:09.916161 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:09.916285 (+ 124us) server_negotiation.cc:407] Connection header received
0924 15:18:09.916356 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:09.916356 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:09.916367 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:09.916419 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:09.916839 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.917277 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.918001 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.918166 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.918215 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.918918 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:09.918919 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:09.918919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:09.918924 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:09.919969 (+ 1045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:09.920257 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:09.920258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:09.920258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:09.920308 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:09.920547 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:09.920548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:09.920548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:09.920683 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:09.920739 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:09.920892 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 15:18:09.920995 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:18:09.921703 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 15:18:09 dist-test-slave-zwbb krb5kdc[1029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 15:18:09 dist-test-slave-zwbb krb5kdc[1029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:09.943607 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.936926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34762 (local address 127.31.12.126:39359)
0924 15:18:09.936984 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:09.936986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:09.937013 (+ 27us) server_negotiation.cc:407] Connection header received
0924 15:18:09.937136 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:09.937137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:09.937147 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:09.937257 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:09.937864 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.938312 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.939045 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.939247 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.939330 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.941498 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:09.941500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:09.941500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:09.941507 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:09.942641 (+ 1134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:09.942934 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:09.942935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:09.942935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:09.942983 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:09.943139 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:09.943139 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:09.943140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:09.943232 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:09.943307 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:09.943405 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 15:18:09.943484 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 15:18:09.945132 1074 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:18:09.946827 1074 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:18:09.953182 1585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.948867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:32925 (local address 127.0.0.1:56086)
0924 15:18:09.949224 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:09.949234 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:09.949283 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:09.949515 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:09.949518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:09.949520 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:09.949859 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.949861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.951094 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.951095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.952069 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.952071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.952936 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.952937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.952993 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.953003 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 15:18:09.953056 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 15:18:09.953090 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":4,"spinlock_wait_cycles":768,"thread_start_us":58,"threads_started":1}
I20260924 15:18:09.953336 1583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.948867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:36935 (local address 127.0.0.1:35062)
0924 15:18:09.949029 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:09.949066 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:09.949126 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:09.949589 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:09.949590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:09.949591 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:09.949653 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.949654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.950990 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.950991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.952283 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.952286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.953142 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.953143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.953197 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.953200 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:18:09.953253 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 15:18:09.953288 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":53,"thread_start_us":53,"threads_started":1}
I20260924 15:18:09.953219 1584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.949007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56086 (local address 127.31.12.66:32925)
0924 15:18:09.949193 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:09.949198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:09.949298 (+ 100us) server_negotiation.cc:407] Connection header received
0924 15:18:09.949343 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:09.949345 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:09.949362 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:09.949529 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:09.949944 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.951003 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.952154 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.952855 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.952932 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.952990 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:09.953072 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 15:18:09.953117 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260924 15:18:09.953639 1586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.949191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35062 (local address 127.31.12.67:36935)
0924 15:18:09.949379 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:09.949383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:09.949399 (+ 16us) server_negotiation.cc:407] Connection header received
0924 15:18:09.949468 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:09.949470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:09.949486 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:09.949733 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:09.949789 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.950914 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.952367 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.953063 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.953128 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.953179 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:09.953448 (+ 269us) server_negotiation.cc:299] Negotiation successful
0924 15:18:09.953520 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":91,"threads_started":1}
I20260924 15:18:09.954053 1359 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6dbae89d504099a6279027009d2f48 (DEFAULT_TABLE table=test-table [id=256ce366d3f0430ea10234d04ea5010e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:09.954146 1501 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6dbae89d504099a6279027009d2f48 (DEFAULT_TABLE table=test-table [id=256ce366d3f0430ea10234d04ea5010e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:09.954408 1501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6dbae89d504099a6279027009d2f48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:09.954409 1359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6dbae89d504099a6279027009d2f48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:09.954806 1582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.949086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:42361 (local address 127.0.0.1:56976)
0924 15:18:09.949605 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:09.949615 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:09.949693 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:09.949857 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:09.949858 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:09.949860 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:09.949904 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.949905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.950865 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.950866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.952283 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.952286 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.954560 (+ 2274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.954561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.954626 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.954629 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:18:09.954704 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 15:18:09.954741 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":89,"thread_start_us":38,"threads_started":1}
I20260924 15:18:09.954851 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.949210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56976 (local address 127.31.12.65:42361)
0924 15:18:09.949558 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:09.949563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:09.949685 (+ 122us) server_negotiation.cc:407] Connection header received
0924 15:18:09.949760 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:09.949762 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:09.949778 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:09.949858 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:09.949960 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.950777 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.953220 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.954419 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.954508 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.954562 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:09.954722 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 15:18:09.954759 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":95,"threads_started":1}
I20260924 15:18:09.955688 1215 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6dbae89d504099a6279027009d2f48 (DEFAULT_TABLE table=test-table [id=256ce366d3f0430ea10234d04ea5010e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:09.956018 1215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6dbae89d504099a6279027009d2f48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:09.958204 1589 tablet_bootstrap.cc:492] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2: Bootstrap starting.
I20260924 15:18:09.958897 1589 tablet_bootstrap.cc:654] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:09.959231 1589 log.cc:824] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:09.959627 1590 tablet_bootstrap.cc:492] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d: Bootstrap starting.
I20260924 15:18:09.959985 1589 tablet_bootstrap.cc:492] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2: No bootstrap required, opened a new log
I20260924 15:18:09.960054 1589 ts_tablet_manager.cc:1397] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:18:09.960279 1590 tablet_bootstrap.cc:654] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:09.960609 1590 log.cc:824] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:09.960929 1588 tablet_bootstrap.cc:492] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7: Bootstrap starting.
I20260924 15:18:09.961303 1590 tablet_bootstrap.cc:492] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d: No bootstrap required, opened a new log
I20260924 15:18:09.961385 1590 ts_tablet_manager.cc:1397] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:18:09.961784 1588 tablet_bootstrap.cc:654] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:09.962105 1588 log.cc:824] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:09.962337 1589 raft_consensus.cc:348] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } } peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } }
I20260924 15:18:09.962503 1589 raft_consensus.cc:374] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:09.962558 1589 raft_consensus.cc:729] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4142cfb5fb54ed5aa8ceaf502688ae2, State: Initialized, Role: FOLLOWER
I20260924 15:18:09.962698 1589 consensus_queue.cc:260] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } } peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } }
I20260924 15:18:09.962968 1589 ts_tablet_manager.cc:1428] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:18:09.962966 1588 tablet_bootstrap.cc:492] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7: No bootstrap required, opened a new log
I20260924 15:18:09.963038 1588 ts_tablet_manager.cc:1397] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:18:09.963096 1425 heartbeater.cc:503] Master 127.31.12.126:39359 was elected leader, sending a full tablet report...
I20260924 15:18:09.963275 1590 raft_consensus.cc:348] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } } peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } }
I20260924 15:18:09.963428 1590 raft_consensus.cc:374] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:09.963481 1590 raft_consensus.cc:729] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a280bc8a849478da774f8bae065653d, State: Initialized, Role: FOLLOWER
I20260924 15:18:09.963596 1590 consensus_queue.cc:260] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } } peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } }
I20260924 15:18:09.963868 1590 ts_tablet_manager.cc:1428] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 15:18:09.963903 1282 heartbeater.cc:503] Master 127.31.12.126:39359 was elected leader, sending a full tablet report...
I20260924 15:18:09.965315 1588 raft_consensus.cc:348] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } } peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } }
I20260924 15:18:09.965435 1588 raft_consensus.cc:374] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:09.965535 1588 raft_consensus.cc:729] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa584d948dc949f5b4f28e1e29f46da7, State: Initialized, Role: FOLLOWER
I20260924 15:18:09.965646 1588 consensus_queue.cc:260] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } } peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } }
I20260924 15:18:09.965921 1588 ts_tablet_manager.cc:1428] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 15:18:09.965942 1567 heartbeater.cc:503] Master 127.31.12.126:39359 was elected leader, sending a full tablet report...
I20260924 15:18:09.970844 1595 raft_consensus.cc:482] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:18:09.970912 1595 raft_consensus.cc:504] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } } peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } }
I20260924 15:18:09.971206 1595 leader_election.cc:288] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa584d948dc949f5b4f28e1e29f46da7 (127.31.12.67:36935), e4142cfb5fb54ed5aa8ceaf502688ae2 (127.31.12.66:32925)
I20260924 15:18:09.974044 1174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.971369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:36935 (local address 127.31.12.65:42091)
0924 15:18:09.971408 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:09.971419 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:09.971457 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:09.971641 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:09.971643 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:09.971645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:09.971709 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.971711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.972380 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.972381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.973158 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.973160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.973794 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.973795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.973875 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.973879 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:18:09.973951 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 15:18:09.973985 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":26}
I20260924 15:18:09.974056 1586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.971464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:42091 (local address 127.31.12.67:36935)
0924 15:18:09.971514 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:09.971516 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:09.971524 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:18:09.971544 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:09.971545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:09.971553 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:09.971623 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:09.971782 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.972308 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.973234 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.973724 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.973779 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.973785 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:09.973949 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 15:18:09.973991 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 15:18:09.974221 1597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.971369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:32925 (local address 127.31.12.65:39485)
0924 15:18:09.971635 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:09.971644 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:09.971700 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:09.971809 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:09.971810 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:09.971811 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:09.971871 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.971873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.972466 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.972467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.973313 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:09.973315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.973947 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.973948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:09.974027 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.974031 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:18:09.974093 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 15:18:09.974136 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":132,"thread_start_us":92,"threads_started":1}
I20260924 15:18:09.974231 1584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:09.971419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:39485 (local address 127.31.12.66:32925)
0924 15:18:09.971459 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:09.971461 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:09.971689 (+ 228us) server_negotiation.cc:407] Connection header received
0924 15:18:09.971738 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:09.971738 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:09.971748 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:09.971805 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:09.971930 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.972386 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.973393 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:09.973839 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:09.973910 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:09.973919 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:09.974110 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 15:18:09.974153 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:18:09.974376 1515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6dbae89d504099a6279027009d2f48" candidate_uuid: "1a280bc8a849478da774f8bae065653d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa584d948dc949f5b4f28e1e29f46da7" is_pre_election: true
I20260924 15:18:09.974505 1515 raft_consensus.cc:2454] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a280bc8a849478da774f8bae065653d in term 0.
I20260924 15:18:09.974692 1169 leader_election.cc:302] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a280bc8a849478da774f8bae065653d, fa584d948dc949f5b4f28e1e29f46da7; no voters:
I20260924 15:18:09.974749 1379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6dbae89d504099a6279027009d2f48" candidate_uuid: "1a280bc8a849478da774f8bae065653d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" is_pre_election: true
I20260924 15:18:09.974822 1595 raft_consensus.cc:2781] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:18:09.974872 1595 raft_consensus.cc:482] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:18:09.974881 1379 raft_consensus.cc:2454] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a280bc8a849478da774f8bae065653d in term 0.
I20260924 15:18:09.974903 1595 raft_consensus.cc:3037] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:09.975581 1595 raft_consensus.cc:504] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } } peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } }
I20260924 15:18:09.975728 1595 leader_election.cc:288] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [CANDIDATE]: Term 1 election: Requested vote from peers fa584d948dc949f5b4f28e1e29f46da7 (127.31.12.67:36935), e4142cfb5fb54ed5aa8ceaf502688ae2 (127.31.12.66:32925)
I20260924 15:18:09.975862 1379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6dbae89d504099a6279027009d2f48" candidate_uuid: "1a280bc8a849478da774f8bae065653d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2"
I20260924 15:18:09.975956 1379 raft_consensus.cc:3037] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:09.976651 1379 raft_consensus.cc:2454] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a280bc8a849478da774f8bae065653d in term 1.
I20260924 15:18:09.976770 1515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6dbae89d504099a6279027009d2f48" candidate_uuid: "1a280bc8a849478da774f8bae065653d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa584d948dc949f5b4f28e1e29f46da7"
I20260924 15:18:09.976852 1515 raft_consensus.cc:3037] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:09.977260 1168 leader_election.cc:302] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a280bc8a849478da774f8bae065653d, e4142cfb5fb54ed5aa8ceaf502688ae2; no voters:
I20260924 15:18:09.977422 1595 raft_consensus.cc:2781] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:09.977597 1595 raft_consensus.cc:686] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [term 1 LEADER]: Becoming Leader. State: Replica: 1a280bc8a849478da774f8bae065653d, State: Running, Role: LEADER
I20260924 15:18:09.977614 1515 raft_consensus.cc:2454] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a280bc8a849478da774f8bae065653d in term 1.
I20260924 15:18:09.977689 1595 consensus_queue.cc:237] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } } peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } }
I20260924 15:18:09.978479 1075 catalog_manager.cc:5495] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a280bc8a849478da774f8bae065653d (127.31.12.65). New cstate: current_term: 1 leader_uuid: "1a280bc8a849478da774f8bae065653d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } health_report { overall_health: UNKNOWN } } }
W20260924 15:18:10.002514 1427 tablet.cc:2369] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 15:18:10 dist-test-slave-zwbb krb5kdc[1029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263089, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:10.008662 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:10.002210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34778 (local address 127.31.12.126:39359)
0924 15:18:10.002302 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:10.002304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:10.002375 (+ 71us) server_negotiation.cc:407] Connection header received
0924 15:18:10.002408 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:10.002409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:10.002421 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:10.002494 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:10.003033 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.003517 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.004174 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.004349 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.004398 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:10.006421 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:10.006423 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:10.006423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:10.006429 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:10.007720 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:10.008071 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:10.008071 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:10.008072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:10.008115 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:10.008280 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:10.008280 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:10.008280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:10.008368 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:10.008414 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:10.008534 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 15:18:10.008598 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 15:18:10.008817 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:10.002121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:39359 (local address 127.0.0.1:34778)
0924 15:18:10.002285 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:10.002296 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:10.002369 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:10.002534 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:10.002535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:10.002902 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:10.002952 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:10.002954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.003588 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.003589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:10.004104 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:10.004105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.004412 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.004413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:10.004458 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:10.004943 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:10.004948 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:10.006348 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:10.007790 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:10.007797 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:10.007799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:10.008012 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:10.008176 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:10.008177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:10.008177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:10.008232 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:10.008417 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:10.008423 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:10.008471 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 15:18:10.008520 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 15:18:10.008674 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":73,"thread_start_us":76,"threads_started":1}
I20260924 15:18:10.010848 1075 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:34778:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 15:18:10.011075 1075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:18:10.012622 1215 tablet_service.cc:1467] Processing CreateTablet for tablet 2202240306c94af09ab868b4ee8f3710 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37f7f205815e44ab89a96cbf87fc48cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:18:10.012645 1501 tablet_service.cc:1467] Processing CreateTablet for tablet 2202240306c94af09ab868b4ee8f3710 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37f7f205815e44ab89a96cbf87fc48cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:18:10.012705 1359 tablet_service.cc:1467] Processing CreateTablet for tablet 2202240306c94af09ab868b4ee8f3710 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=37f7f205815e44ab89a96cbf87fc48cc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:18:10.012810 1215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2202240306c94af09ab868b4ee8f3710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:10.012812 1501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2202240306c94af09ab868b4ee8f3710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:10.012847 1359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2202240306c94af09ab868b4ee8f3710. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:10.014148 1590 tablet_bootstrap.cc:492] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d: Bootstrap starting.
I20260924 15:18:10.014309 1589 tablet_bootstrap.cc:492] T 2202240306c94af09ab868b4ee8f3710 P e4142cfb5fb54ed5aa8ceaf502688ae2: Bootstrap starting.
I20260924 15:18:10.014309 1588 tablet_bootstrap.cc:492] T 2202240306c94af09ab868b4ee8f3710 P fa584d948dc949f5b4f28e1e29f46da7: Bootstrap starting.
I20260924 15:18:10.014562 1590 tablet_bootstrap.cc:654] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:10.014897 1588 tablet_bootstrap.cc:654] T 2202240306c94af09ab868b4ee8f3710 P fa584d948dc949f5b4f28e1e29f46da7: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:10.015028 1589 tablet_bootstrap.cc:654] T 2202240306c94af09ab868b4ee8f3710 P e4142cfb5fb54ed5aa8ceaf502688ae2: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:10.015578 1588 tablet_bootstrap.cc:492] T 2202240306c94af09ab868b4ee8f3710 P fa584d948dc949f5b4f28e1e29f46da7: No bootstrap required, opened a new log
I20260924 15:18:10.015630 1588 ts_tablet_manager.cc:1397] T 2202240306c94af09ab868b4ee8f3710 P fa584d948dc949f5b4f28e1e29f46da7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 15:18:10.015686 1589 tablet_bootstrap.cc:492] T 2202240306c94af09ab868b4ee8f3710 P e4142cfb5fb54ed5aa8ceaf502688ae2: No bootstrap required, opened a new log
I20260924 15:18:10.015728 1589 ts_tablet_manager.cc:1397] T 2202240306c94af09ab868b4ee8f3710 P e4142cfb5fb54ed5aa8ceaf502688ae2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:10.015799 1588 raft_consensus.cc:348] T 2202240306c94af09ab868b4ee8f3710 P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:10.015897 1588 raft_consensus.cc:374] T 2202240306c94af09ab868b4ee8f3710 P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:10.015856 1589 raft_consensus.cc:348] T 2202240306c94af09ab868b4ee8f3710 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:10.015921 1588 raft_consensus.cc:729] T 2202240306c94af09ab868b4ee8f3710 P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa584d948dc949f5b4f28e1e29f46da7, State: Initialized, Role: FOLLOWER
I20260924 15:18:10.015924 1589 raft_consensus.cc:374] T 2202240306c94af09ab868b4ee8f3710 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:10.015926 1590 tablet_bootstrap.cc:492] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d: No bootstrap required, opened a new log
I20260924 15:18:10.015966 1589 raft_consensus.cc:729] T 2202240306c94af09ab868b4ee8f3710 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4142cfb5fb54ed5aa8ceaf502688ae2, State: Initialized, Role: FOLLOWER
I20260924 15:18:10.015978 1590 ts_tablet_manager.cc:1397] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:18:10.015987 1588 consensus_queue.cc:260] T 2202240306c94af09ab868b4ee8f3710 P fa584d948dc949f5b4f28e1e29f46da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:10.016012 1589 consensus_queue.cc:260] T 2202240306c94af09ab868b4ee8f3710 P e4142cfb5fb54ed5aa8ceaf502688ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:10.016098 1589 ts_tablet_manager.cc:1428] T 2202240306c94af09ab868b4ee8f3710 P e4142cfb5fb54ed5aa8ceaf502688ae2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:10.016098 1588 ts_tablet_manager.cc:1428] T 2202240306c94af09ab868b4ee8f3710 P fa584d948dc949f5b4f28e1e29f46da7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:10.016124 1590 raft_consensus.cc:348] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:10.016144 1596 tablet_replica.cc:439] T 2202240306c94af09ab868b4ee8f3710 P fa584d948dc949f5b4f28e1e29f46da7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } } }
I20260924 15:18:10.016202 1590 raft_consensus.cc:374] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:10.016237 1590 raft_consensus.cc:729] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a280bc8a849478da774f8bae065653d, State: Initialized, Role: FOLLOWER
I20260924 15:18:10.016240 1596 tablet_replica.cc:442] T 2202240306c94af09ab868b4ee8f3710 P fa584d948dc949f5b4f28e1e29f46da7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:18:10.016294 1590 consensus_queue.cc:260] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:10.016402 1590 ts_tablet_manager.cc:1428] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:10.016391 1594 tablet_replica.cc:439] T 2202240306c94af09ab868b4ee8f3710 P e4142cfb5fb54ed5aa8ceaf502688ae2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } } }
I20260924 15:18:10.016444 1595 tablet_replica.cc:439] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } } }
I20260924 15:18:10.016505 1595 tablet_replica.cc:442] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:18:10.016518 1594 tablet_replica.cc:442] T 2202240306c94af09ab868b4ee8f3710 P e4142cfb5fb54ed5aa8ceaf502688ae2: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 15:18:10.104135 1283 tablet.cc:2369] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:18:10.163153 1568 tablet.cc:2369] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:18:10.205941 1595 raft_consensus.cc:482] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:18:10.206081 1595 raft_consensus.cc:504] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:10.206285 1595 leader_election.cc:288] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4142cfb5fb54ed5aa8ceaf502688ae2 (127.31.12.66:32925), fa584d948dc949f5b4f28e1e29f46da7 (127.31.12.67:36935)
I20260924 15:18:10.206543 1379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2202240306c94af09ab868b4ee8f3710" candidate_uuid: "1a280bc8a849478da774f8bae065653d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" is_pre_election: true
I20260924 15:18:10.206673 1379 raft_consensus.cc:2454] T 2202240306c94af09ab868b4ee8f3710 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a280bc8a849478da774f8bae065653d in term 0.
I20260924 15:18:10.206678 1515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2202240306c94af09ab868b4ee8f3710" candidate_uuid: "1a280bc8a849478da774f8bae065653d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa584d948dc949f5b4f28e1e29f46da7" is_pre_election: true
I20260924 15:18:10.206786 1515 raft_consensus.cc:2454] T 2202240306c94af09ab868b4ee8f3710 P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a280bc8a849478da774f8bae065653d in term 0.
I20260924 15:18:10.206843 1168 leader_election.cc:302] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a280bc8a849478da774f8bae065653d, e4142cfb5fb54ed5aa8ceaf502688ae2; no voters:
I20260924 15:18:10.206964 1595 raft_consensus.cc:2781] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:18:10.206998 1595 raft_consensus.cc:482] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:18:10.207032 1595 raft_consensus.cc:3037] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:10.207849 1595 raft_consensus.cc:504] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:10.207998 1595 leader_election.cc:288] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d [CANDIDATE]: Term 1 election: Requested vote from peers e4142cfb5fb54ed5aa8ceaf502688ae2 (127.31.12.66:32925), fa584d948dc949f5b4f28e1e29f46da7 (127.31.12.67:36935)
I20260924 15:18:10.208161 1379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2202240306c94af09ab868b4ee8f3710" candidate_uuid: "1a280bc8a849478da774f8bae065653d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2"
I20260924 15:18:10.208186 1515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2202240306c94af09ab868b4ee8f3710" candidate_uuid: "1a280bc8a849478da774f8bae065653d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa584d948dc949f5b4f28e1e29f46da7"
I20260924 15:18:10.208240 1379 raft_consensus.cc:3037] T 2202240306c94af09ab868b4ee8f3710 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:10.208244 1515 raft_consensus.cc:3037] T 2202240306c94af09ab868b4ee8f3710 P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:10.208976 1379 raft_consensus.cc:2454] T 2202240306c94af09ab868b4ee8f3710 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a280bc8a849478da774f8bae065653d in term 1.
I20260924 15:18:10.208983 1515 raft_consensus.cc:2454] T 2202240306c94af09ab868b4ee8f3710 P fa584d948dc949f5b4f28e1e29f46da7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a280bc8a849478da774f8bae065653d in term 1.
I20260924 15:18:10.209160 1169 leader_election.cc:302] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a280bc8a849478da774f8bae065653d, fa584d948dc949f5b4f28e1e29f46da7; no voters:
I20260924 15:18:10.209293 1595 raft_consensus.cc:2781] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:10.209367 1595 raft_consensus.cc:686] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d [term 1 LEADER]: Becoming Leader. State: Replica: 1a280bc8a849478da774f8bae065653d, State: Running, Role: LEADER
I20260924 15:18:10.209424 1595 consensus_queue.cc:237] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:10.209765 1598 tablet_replica.cc:439] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d: TxnStatusTablet state changed. Reason: New leader 1a280bc8a849478da774f8bae065653d. Latest consensus state: current_term: 1 leader_uuid: "1a280bc8a849478da774f8bae065653d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } } }
I20260924 15:18:10.209828 1598 tablet_replica.cc:442] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:18:10.209992 1610 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 15:18:10.210127 1610 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 15:18:10.210093 1073 catalog_manager.cc:5495] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a280bc8a849478da774f8bae065653d (127.31.12.65). New cstate: current_term: 1 leader_uuid: "1a280bc8a849478da774f8bae065653d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } health_report { overall_health: UNKNOWN } } }
I20260924 15:18:10.245517 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:10.242681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:42361 (local address 127.0.0.1:56990)
0924 15:18:10.242760 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:10.242773 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:10.242827 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:10.243054 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:10.243055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:10.243056 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:18:10.243117 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:10.243119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.243840 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.243841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:10.244429 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:10.244431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.244891 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.244892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:10.245000 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:10.245020 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:18:10.245315 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:18:10.245325 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 15:18:10.245373 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 15:18:10.245429 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 15:18:10.245505 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:10.242752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56990 (local address 127.31.12.65:42361)
0924 15:18:10.242805 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:10.242808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:10.242819 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:18:10.242919 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:10.242920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:10.242948 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:10.243046 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:18:10.243189 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.243760 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.244519 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.244794 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.244888 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:10.245137 (+ 249us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:18:10.245248 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:18:10.245337 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:10.245380 (+ 43us) server_negotiation.cc:299] Negotiation successful
0924 15:18:10.245403 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 15:18:10.247464 1515 raft_consensus.cc:1261] T 2202240306c94af09ab868b4ee8f3710 P fa584d948dc949f5b4f28e1e29f46da7 [term 1 FOLLOWER]: Refusing update from remote peer 1a280bc8a849478da774f8bae065653d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:10.247489 1379 raft_consensus.cc:1261] T 2202240306c94af09ab868b4ee8f3710 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 1 FOLLOWER]: Refusing update from remote peer 1a280bc8a849478da774f8bae065653d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:10.247781 1595 consensus_queue.cc:1035] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:10.247901 1598 consensus_queue.cc:1035] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:10.249115 1594 tablet_replica.cc:439] T 2202240306c94af09ab868b4ee8f3710 P e4142cfb5fb54ed5aa8ceaf502688ae2: TxnStatusTablet state changed. Reason: New leader 1a280bc8a849478da774f8bae065653d. Latest consensus state: current_term: 1 leader_uuid: "1a280bc8a849478da774f8bae065653d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } } }
I20260924 15:18:10.249240 1594 tablet_replica.cc:442] T 2202240306c94af09ab868b4ee8f3710 P e4142cfb5fb54ed5aa8ceaf502688ae2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:18:10.249378 1596 tablet_replica.cc:439] T 2202240306c94af09ab868b4ee8f3710 P fa584d948dc949f5b4f28e1e29f46da7: TxnStatusTablet state changed. Reason: New leader 1a280bc8a849478da774f8bae065653d. Latest consensus state: current_term: 1 leader_uuid: "1a280bc8a849478da774f8bae065653d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } } }
I20260924 15:18:10.249503 1596 tablet_replica.cc:442] T 2202240306c94af09ab868b4ee8f3710 P fa584d948dc949f5b4f28e1e29f46da7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:18:10.250399 1598 tablet_replica.cc:439] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a280bc8a849478da774f8bae065653d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } } }
I20260924 15:18:10.250402 1594 tablet_replica.cc:439] T 2202240306c94af09ab868b4ee8f3710 P e4142cfb5fb54ed5aa8ceaf502688ae2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a280bc8a849478da774f8bae065653d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } } }
I20260924 15:18:10.250487 1598 tablet_replica.cc:442] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:18:10.250490 1594 tablet_replica.cc:442] T 2202240306c94af09ab868b4ee8f3710 P e4142cfb5fb54ed5aa8ceaf502688ae2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:18:10.250824 1595 tablet_replica.cc:439] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1a280bc8a849478da774f8bae065653d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } } }
I20260924 15:18:10.250898 1595 tablet_replica.cc:442] T 2202240306c94af09ab868b4ee8f3710 P 1a280bc8a849478da774f8bae065653d: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:18:10.250974 1613 mvcc.cc:205] Tried to move back new op lower bound from 7332917617650876416 to 7332917617498365952. Current Snapshot: MvccSnapshot[applied={T|T < 7332917617650876416 or (T in {7332917617650876416})}]
I20260924 15:18:10.251370 1596 tablet_replica.cc:439] T 2202240306c94af09ab868b4ee8f3710 P fa584d948dc949f5b4f28e1e29f46da7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1a280bc8a849478da774f8bae065653d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } } }
I20260924 15:18:10.251459 1596 tablet_replica.cc:442] T 2202240306c94af09ab868b4ee8f3710 P fa584d948dc949f5b4f28e1e29f46da7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:18:10.251818 1615 mvcc.cc:205] Tried to move back new op lower bound from 7332917617650876416 to 7332917617498365952. Current Snapshot: MvccSnapshot[applied={T|T < 7332917617650876416}]
I20260924 15:18:10.252460 1614 mvcc.cc:205] Tried to move back new op lower bound from 7332917617650876416 to 7332917617498365952. Current Snapshot: MvccSnapshot[applied={T|T < 7332917617650876416}]
I20260924 15:18:10.254930 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:10.252595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57006 (local address 127.31.12.65:42361)
0924 15:18:10.252641 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:10.252643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:10.252651 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:18:10.252738 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:10.252740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:10.252749 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:10.252791 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:18:10.252973 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.253410 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.254211 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.254415 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.254497 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:10.254670 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:18:10.254700 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:18:10.254766 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:10.254827 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 15:18:10.254845 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:18:10.259977 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:10.257394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:41441 (local address 127.31.12.65:42361)
0924 15:18:10.257463 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:10.257465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:10.257549 (+ 84us) server_negotiation.cc:407] Connection header received
0924 15:18:10.257618 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:10.257618 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:10.257625 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:10.257668 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:18:10.257824 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.258232 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.259334 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.259505 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.259548 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:10.259760 (+ 212us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:18:10.259779 (+ 19us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:18:10.259815 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:10.259891 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 15:18:10.259905 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:18:10.260135 1625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:10.257312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:42361 (local address 127.31.12.65:41441)
0924 15:18:10.257494 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:10.257505 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:10.257564 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:10.257690 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:10.257691 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:10.257693 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:18:10.257769 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:10.257770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.258298 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.258299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:10.259251 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:10.259253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.259570 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.259571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:10.259659 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:10.259674 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:18:10.259837 (+ 163us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:18:10.259839 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:18:10.259890 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 15:18:10.259935 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20260924 15:18:10.262259 1379 raft_consensus.cc:1261] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 1 FOLLOWER]: Refusing update from remote peer 1a280bc8a849478da774f8bae065653d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:10.262272 1515 raft_consensus.cc:1261] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7 [term 1 FOLLOWER]: Refusing update from remote peer 1a280bc8a849478da774f8bae065653d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:10.262467 1595 consensus_queue.cc:1035] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:10.262547 1626 consensus_queue.cc:1035] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:10.275028 1073 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:18:10.275125 1073 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:18:10.275938 1073 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788: Sending DeleteTablet for 3 replicas of tablet 0f6dbae89d504099a6279027009d2f48
I20260924 15:18:10.276274 1214 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f6dbae89d504099a6279027009d2f48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:10 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:56976
I20260924 15:18:10.276288 1501 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f6dbae89d504099a6279027009d2f48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:10 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:35062
I20260924 15:18:10.276338 1359 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f6dbae89d504099a6279027009d2f48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:10 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:56086
I20260924 15:18:10.276489 1632 tablet_replica.cc:330] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2: stopping tablet replica
I20260924 15:18:10.276571 1632 raft_consensus.cc:2229] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:18:10.276611 1632 raft_consensus.cc:2258] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:10.276757 1631 tablet_replica.cc:330] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7: stopping tablet replica
I20260924 15:18:10.276842 1631 raft_consensus.cc:2229] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:18:10.276880 1631 raft_consensus.cc:2258] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:10.277019 1630 tablet_replica.cc:330] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d: stopping tablet replica
I20260924 15:18:10.276968 1073 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:18:10.277108 1073 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:18:10.277292 1632 ts_tablet_manager.cc:1905] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:10.277313 1630 raft_consensus.cc:2229] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:18:10.277639 1630 raft_consensus.cc:2258] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:10.278368 1359 tablet_service.cc:1467] Processing CreateTablet for tablet 1897fb8efdb94c35986efe2518e1f0e3 (DEFAULT_TABLE table=test-table [id=503800fd9f94409d9377bd1cda029de1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:10.278475 1359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1897fb8efdb94c35986efe2518e1f0e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:10.278605 1630 ts_tablet_manager.cc:1905] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:10.279001 1632 ts_tablet_manager.cc:1918] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:18:10.279088 1632 log.cc:1196] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/0f6dbae89d504099a6279027009d2f48
I20260924 15:18:10.279388 1632 ts_tablet_manager.cc:1939] T 0f6dbae89d504099a6279027009d2f48 P e4142cfb5fb54ed5aa8ceaf502688ae2: Deleting consensus metadata
I20260924 15:18:10.279666 1589 tablet_bootstrap.cc:492] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2: Bootstrap starting.
I20260924 15:18:10.279798 1501 tablet_service.cc:1467] Processing CreateTablet for tablet 1897fb8efdb94c35986efe2518e1f0e3 (DEFAULT_TABLE table=test-table [id=503800fd9f94409d9377bd1cda029de1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:10.279919 1501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1897fb8efdb94c35986efe2518e1f0e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:10.280061 1589 tablet_bootstrap.cc:654] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:10.280751 1060 catalog_manager.cc:4841] TS e4142cfb5fb54ed5aa8ceaf502688ae2 (127.31.12.66:32925): tablet 0f6dbae89d504099a6279027009d2f48 (table test-table [id=256ce366d3f0430ea10234d04ea5010e]) successfully deleted
I20260924 15:18:10.281145 1589 tablet_bootstrap.cc:492] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2: No bootstrap required, opened a new log
I20260924 15:18:10.281201 1589 ts_tablet_manager.cc:1397] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:18:10.281347 1589 raft_consensus.cc:348] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:10.281395 1588 tablet_bootstrap.cc:492] T 1897fb8efdb94c35986efe2518e1f0e3 P fa584d948dc949f5b4f28e1e29f46da7: Bootstrap starting.
I20260924 15:18:10.281428 1589 raft_consensus.cc:374] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:10.281472 1589 raft_consensus.cc:729] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4142cfb5fb54ed5aa8ceaf502688ae2, State: Initialized, Role: FOLLOWER
I20260924 15:18:10.281515 1589 consensus_queue.cc:260] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:10.281389 1630 ts_tablet_manager.cc:1918] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:18:10.281627 1589 ts_tablet_manager.cc:1428] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:10.281700 1630 log.cc:1196] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-0/wal/wals/0f6dbae89d504099a6279027009d2f48
I20260924 15:18:10.281798 1588 tablet_bootstrap.cc:654] T 1897fb8efdb94c35986efe2518e1f0e3 P fa584d948dc949f5b4f28e1e29f46da7: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:10.282063 1630 ts_tablet_manager.cc:1939] T 0f6dbae89d504099a6279027009d2f48 P 1a280bc8a849478da774f8bae065653d: Deleting consensus metadata
I20260924 15:18:10.282358 1214 tablet_service.cc:1467] Processing CreateTablet for tablet 1897fb8efdb94c35986efe2518e1f0e3 (DEFAULT_TABLE table=test-table [id=503800fd9f94409d9377bd1cda029de1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:10.282466 1214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1897fb8efdb94c35986efe2518e1f0e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:10.282562 1061 catalog_manager.cc:4841] TS 1a280bc8a849478da774f8bae065653d (127.31.12.65:42361): tablet 0f6dbae89d504099a6279027009d2f48 (table test-table [id=256ce366d3f0430ea10234d04ea5010e]) successfully deleted
I20260924 15:18:10.283435 1590 tablet_bootstrap.cc:492] T 1897fb8efdb94c35986efe2518e1f0e3 P 1a280bc8a849478da774f8bae065653d: Bootstrap starting.
I20260924 15:18:10.283758 1588 tablet_bootstrap.cc:492] T 1897fb8efdb94c35986efe2518e1f0e3 P fa584d948dc949f5b4f28e1e29f46da7: No bootstrap required, opened a new log
I20260924 15:18:10.283790 1588 ts_tablet_manager.cc:1397] T 1897fb8efdb94c35986efe2518e1f0e3 P fa584d948dc949f5b4f28e1e29f46da7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:18:10.283804 1590 tablet_bootstrap.cc:654] T 1897fb8efdb94c35986efe2518e1f0e3 P 1a280bc8a849478da774f8bae065653d: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:10.283895 1588 raft_consensus.cc:348] T 1897fb8efdb94c35986efe2518e1f0e3 P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:10.283936 1588 raft_consensus.cc:374] T 1897fb8efdb94c35986efe2518e1f0e3 P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:10.283949 1588 raft_consensus.cc:729] T 1897fb8efdb94c35986efe2518e1f0e3 P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa584d948dc949f5b4f28e1e29f46da7, State: Initialized, Role: FOLLOWER
I20260924 15:18:10.284018 1588 consensus_queue.cc:260] T 1897fb8efdb94c35986efe2518e1f0e3 P fa584d948dc949f5b4f28e1e29f46da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:10.284112 1588 ts_tablet_manager.cc:1428] T 1897fb8efdb94c35986efe2518e1f0e3 P fa584d948dc949f5b4f28e1e29f46da7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:10.284480 1590 tablet_bootstrap.cc:492] T 1897fb8efdb94c35986efe2518e1f0e3 P 1a280bc8a849478da774f8bae065653d: No bootstrap required, opened a new log
I20260924 15:18:10.284529 1590 ts_tablet_manager.cc:1397] T 1897fb8efdb94c35986efe2518e1f0e3 P 1a280bc8a849478da774f8bae065653d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:10.284662 1590 raft_consensus.cc:348] T 1897fb8efdb94c35986efe2518e1f0e3 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:10.284722 1590 raft_consensus.cc:374] T 1897fb8efdb94c35986efe2518e1f0e3 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:10.284740 1590 raft_consensus.cc:729] T 1897fb8efdb94c35986efe2518e1f0e3 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a280bc8a849478da774f8bae065653d, State: Initialized, Role: FOLLOWER
I20260924 15:18:10.284776 1590 consensus_queue.cc:260] T 1897fb8efdb94c35986efe2518e1f0e3 P 1a280bc8a849478da774f8bae065653d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:10.284870 1590 ts_tablet_manager.cc:1428] T 1897fb8efdb94c35986efe2518e1f0e3 P 1a280bc8a849478da774f8bae065653d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:10.285689 1631 ts_tablet_manager.cc:1905] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:10.287187 1631 ts_tablet_manager.cc:1918] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:18:10.287276 1631 log.cc:1196] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/0f6dbae89d504099a6279027009d2f48
I20260924 15:18:10.287525 1631 ts_tablet_manager.cc:1939] T 0f6dbae89d504099a6279027009d2f48 P fa584d948dc949f5b4f28e1e29f46da7: Deleting consensus metadata
I20260924 15:18:10.287998 1061 catalog_manager.cc:4841] TS fa584d948dc949f5b4f28e1e29f46da7 (127.31.12.67:36935): tablet 0f6dbae89d504099a6279027009d2f48 (table test-table [id=256ce366d3f0430ea10234d04ea5010e]) successfully deleted
I20260924 15:18:10.357234 1594 raft_consensus.cc:482] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:18:10.357349 1594 raft_consensus.cc:504] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:10.357756 1594 leader_election.cc:288] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a280bc8a849478da774f8bae065653d (127.31.12.65:42361), fa584d948dc949f5b4f28e1e29f46da7 (127.31.12.67:36935)
I20260924 15:18:10.360853 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:10.358007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:35103 (local address 127.31.12.65:42361)
0924 15:18:10.358052 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:10.358054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:10.358104 (+ 50us) server_negotiation.cc:407] Connection header received
0924 15:18:10.358202 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:10.358203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:10.358214 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:10.358287 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:10.358473 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.358920 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.359900 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.360486 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.360561 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:10.360571 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:10.360755 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 15:18:10.360789 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:18:10.360864 1636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:10.357921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:42361 (local address 127.31.12.66:35103)
0924 15:18:10.358029 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:10.358043 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:10.358132 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:10.358294 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:10.358295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:10.358297 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:10.358379 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:10.358381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.359000 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.359001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:10.359806 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:10.359809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.360578 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.360579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:10.360667 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:10.360672 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:18:10.360731 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 15:18:10.360774 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":55,"threads_started":1}
I20260924 15:18:10.361238 1235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1897fb8efdb94c35986efe2518e1f0e3" candidate_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a280bc8a849478da774f8bae065653d" is_pre_election: true
I20260924 15:18:10.361368 1235 raft_consensus.cc:2454] T 1897fb8efdb94c35986efe2518e1f0e3 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4142cfb5fb54ed5aa8ceaf502688ae2 in term 0.
I20260924 15:18:10.361948 1312 leader_election.cc:302] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a280bc8a849478da774f8bae065653d, e4142cfb5fb54ed5aa8ceaf502688ae2; no voters:
I20260924 15:18:10.362036 1586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:10.358160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:42337 (local address 127.31.12.67:36935)
0924 15:18:10.358209 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:10.358211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:10.358269 (+ 58us) server_negotiation.cc:407] Connection header received
0924 15:18:10.358376 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:10.358377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:10.358386 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:10.358444 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:10.359187 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.359900 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.361207 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.361692 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.361785 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:10.361809 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:10.361943 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 15:18:10.361977 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 15:18:10.362145 1594 raft_consensus.cc:2781] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:18:10.362150 1637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:10.358065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:36935 (local address 127.31.12.66:42337)
0924 15:18:10.358215 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:10.358224 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:10.358290 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:10.359040 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:10.359041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:10.359043 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:10.359101 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:10.359103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.359998 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.359999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:10.361138 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:10.361141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.361784 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.361785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:10.361873 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:10.361878 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:18:10.361932 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 15:18:10.361966 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":47,"threads_started":1}
I20260924 15:18:10.362200 1594 raft_consensus.cc:482] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:18:10.362227 1594 raft_consensus.cc:3037] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:10.362895 1594 raft_consensus.cc:504] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:10.363023 1594 leader_election.cc:288] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [CANDIDATE]: Term 1 election: Requested vote from peers 1a280bc8a849478da774f8bae065653d (127.31.12.65:42361), fa584d948dc949f5b4f28e1e29f46da7 (127.31.12.67:36935)
I20260924 15:18:10.363229 1235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1897fb8efdb94c35986efe2518e1f0e3" candidate_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a280bc8a849478da774f8bae065653d"
I20260924 15:18:10.363214 1515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1897fb8efdb94c35986efe2518e1f0e3" candidate_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa584d948dc949f5b4f28e1e29f46da7" is_pre_election: true
I20260924 15:18:10.363232 1521 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1897fb8efdb94c35986efe2518e1f0e3" candidate_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa584d948dc949f5b4f28e1e29f46da7"
I20260924 15:18:10.363307 1235 raft_consensus.cc:3037] T 1897fb8efdb94c35986efe2518e1f0e3 P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:10.363313 1515 raft_consensus.cc:2454] T 1897fb8efdb94c35986efe2518e1f0e3 P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4142cfb5fb54ed5aa8ceaf502688ae2 in term 0.
I20260924 15:18:10.363888 1235 raft_consensus.cc:2454] T 1897fb8efdb94c35986efe2518e1f0e3 P 1a280bc8a849478da774f8bae065653d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4142cfb5fb54ed5aa8ceaf502688ae2 in term 1.
I20260924 15:18:10.364106 1312 leader_election.cc:302] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a280bc8a849478da774f8bae065653d, e4142cfb5fb54ed5aa8ceaf502688ae2; no voters:
I20260924 15:18:10.364213 1594 raft_consensus.cc:2781] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:10.364393 1594 raft_consensus.cc:686] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 1 LEADER]: Becoming Leader. State: Replica: e4142cfb5fb54ed5aa8ceaf502688ae2, State: Running, Role: LEADER
I20260924 15:18:10.364477 1594 consensus_queue.cc:237] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:10.365342 1073 catalog_manager.cc:5495] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4142cfb5fb54ed5aa8ceaf502688ae2 (127.31.12.66). New cstate: current_term: 1 leader_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } health_report { overall_health: UNKNOWN } } }
I20260924 15:18:10.377421 1584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:10.374850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56098 (local address 127.31.12.66:32925)
0924 15:18:10.374886 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:10.374888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:10.374896 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:18:10.374974 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:10.374975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:10.374995 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:10.375080 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:18:10.375162 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.375911 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.376597 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.376895 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.376963 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:10.377078 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:18:10.377208 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:18:10.377274 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:10.377314 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 15:18:10.377335 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:18:10.382572 1587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:10.380028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:51499 (local address 127.31.12.65:42361)
0924 15:18:10.380074 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:10.380079 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:10.380137 (+ 58us) server_negotiation.cc:407] Connection header received
0924 15:18:10.380206 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:10.380207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:10.380214 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:10.380258 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:18:10.380375 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.380835 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.381939 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.382114 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.382161 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:10.382363 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:18:10.382390 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:18:10.382424 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:10.382508 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 15:18:10.382521 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:18:10.382640 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:10.379933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:42361 (local address 127.31.12.66:51499)
0924 15:18:10.380086 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:10.380097 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:10.380151 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:10.380260 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:10.380261 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:10.380263 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:18:10.380316 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:10.380317 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.380909 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.380909 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:10.381846 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:10.381849 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.382182 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.382183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:10.382266 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:10.382281 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:18:10.382450 (+ 169us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:18:10.382452 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:18:10.382506 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 15:18:10.382547 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":75,"threads_started":1}
I20260924 15:18:10.384452 1515 raft_consensus.cc:3037] T 1897fb8efdb94c35986efe2518e1f0e3 P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:10.384593 1235 raft_consensus.cc:1261] T 1897fb8efdb94c35986efe2518e1f0e3 P 1a280bc8a849478da774f8bae065653d [term 1 FOLLOWER]: Refusing update from remote peer e4142cfb5fb54ed5aa8ceaf502688ae2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:10.384835 1594 consensus_queue.cc:1035] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:10.385465 1515 raft_consensus.cc:1261] T 1897fb8efdb94c35986efe2518e1f0e3 P fa584d948dc949f5b4f28e1e29f46da7 [term 1 FOLLOWER]: Refusing update from remote peer e4142cfb5fb54ed5aa8ceaf502688ae2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:10.385886 1594 consensus_queue.cc:1035] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:10.392396 1584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:10.389829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:40793 (local address 127.31.12.66:32925)
0924 15:18:10.389865 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:10.389867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:10.389889 (+ 22us) server_negotiation.cc:407] Connection header received
0924 15:18:10.389967 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:10.389968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:10.389979 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:10.390024 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:18:10.390184 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.390702 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.391744 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.392021 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.392072 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:10.392224 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:18:10.392249 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:18:10.392284 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:10.392336 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 15:18:10.392352 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:18:10.392453 1625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:10.389794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:32925 (local address 127.31.12.65:40793)
0924 15:18:10.389843 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:10.389855 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:10.389903 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:10.390067 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:10.390068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:10.390069 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:18:10.390110 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:10.390111 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.390784 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.390785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:10.391672 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:10.391675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:10.392092 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:10.392093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:10.392150 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:10.392156 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:18:10.392285 (+ 129us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:18:10.392286 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 15:18:10.392333 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 15:18:10.392374 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 15:18:10.392730 1615 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 15:18:11.912261 1073 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:18:11.912376 1073 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:18:11.913323 1073 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788: Sending DeleteTablet for 3 replicas of tablet 1897fb8efdb94c35986efe2518e1f0e3
I20260924 15:18:11.913630 1501 tablet_service.cc:1514] Processing DeleteTablet for tablet 1897fb8efdb94c35986efe2518e1f0e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:11 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:35062
I20260924 15:18:11.915249 1214 tablet_service.cc:1514] Processing DeleteTablet for tablet 1897fb8efdb94c35986efe2518e1f0e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:11 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:56976
I20260924 15:18:11.915452 1670 tablet_replica.cc:330] T 1897fb8efdb94c35986efe2518e1f0e3 P 1a280bc8a849478da774f8bae065653d: stopping tablet replica
I20260924 15:18:11.915477 1359 tablet_service.cc:1514] Processing DeleteTablet for tablet 1897fb8efdb94c35986efe2518e1f0e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:11 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:56086
I20260924 15:18:11.915524 1670 raft_consensus.cc:2229] T 1897fb8efdb94c35986efe2518e1f0e3 P 1a280bc8a849478da774f8bae065653d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:18:11.915570 1670 raft_consensus.cc:2258] T 1897fb8efdb94c35986efe2518e1f0e3 P 1a280bc8a849478da774f8bae065653d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:11.915838 1670 ts_tablet_manager.cc:1905] T 1897fb8efdb94c35986efe2518e1f0e3 P 1a280bc8a849478da774f8bae065653d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:11.916482 1671 tablet_replica.cc:330] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2: stopping tablet replica
I20260924 15:18:11.916536 1671 raft_consensus.cc:2229] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:18:11.916620 1671 raft_consensus.cc:2258] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:11.917573 1668 tablet_replica.cc:330] T 1897fb8efdb94c35986efe2518e1f0e3 P fa584d948dc949f5b4f28e1e29f46da7: stopping tablet replica
I20260924 15:18:11.917644 1668 raft_consensus.cc:2229] T 1897fb8efdb94c35986efe2518e1f0e3 P fa584d948dc949f5b4f28e1e29f46da7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:18:11.917691 1668 raft_consensus.cc:2258] T 1897fb8efdb94c35986efe2518e1f0e3 P fa584d948dc949f5b4f28e1e29f46da7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:11.917753 1671 ts_tablet_manager.cc:1905] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:11.917940 1668 ts_tablet_manager.cc:1905] T 1897fb8efdb94c35986efe2518e1f0e3 P fa584d948dc949f5b4f28e1e29f46da7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:11.918332 1670 ts_tablet_manager.cc:1918] T 1897fb8efdb94c35986efe2518e1f0e3 P 1a280bc8a849478da774f8bae065653d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 15:18:11.918398 1670 log.cc:1196] T 1897fb8efdb94c35986efe2518e1f0e3 P 1a280bc8a849478da774f8bae065653d: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-0/wal/wals/1897fb8efdb94c35986efe2518e1f0e3
I20260924 15:18:11.918622 1670 ts_tablet_manager.cc:1939] T 1897fb8efdb94c35986efe2518e1f0e3 P 1a280bc8a849478da774f8bae065653d: Deleting consensus metadata
I20260924 15:18:11.918985 1061 catalog_manager.cc:4841] TS 1a280bc8a849478da774f8bae065653d (127.31.12.65:42361): tablet 1897fb8efdb94c35986efe2518e1f0e3 (table test-table [id=503800fd9f94409d9377bd1cda029de1]) successfully deleted
I20260924 15:18:11.919349 1671 ts_tablet_manager.cc:1918] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 15:18:11.919426 1671 log.cc:1196] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/1897fb8efdb94c35986efe2518e1f0e3
I20260924 15:18:11.919658 1671 ts_tablet_manager.cc:1939] T 1897fb8efdb94c35986efe2518e1f0e3 P e4142cfb5fb54ed5aa8ceaf502688ae2: Deleting consensus metadata
I20260924 15:18:11.919684 1668 ts_tablet_manager.cc:1918] T 1897fb8efdb94c35986efe2518e1f0e3 P fa584d948dc949f5b4f28e1e29f46da7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 15:18:11.919739 1668 log.cc:1196] T 1897fb8efdb94c35986efe2518e1f0e3 P fa584d948dc949f5b4f28e1e29f46da7: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/1897fb8efdb94c35986efe2518e1f0e3
I20260924 15:18:11.919965 1668 ts_tablet_manager.cc:1939] T 1897fb8efdb94c35986efe2518e1f0e3 P fa584d948dc949f5b4f28e1e29f46da7: Deleting consensus metadata
I20260924 15:18:11.920272 1060 catalog_manager.cc:4841] TS e4142cfb5fb54ed5aa8ceaf502688ae2 (127.31.12.66:32925): tablet 1897fb8efdb94c35986efe2518e1f0e3 (table test-table [id=503800fd9f94409d9377bd1cda029de1]) successfully deleted
I20260924 15:18:11.920363 1061 catalog_manager.cc:4841] TS fa584d948dc949f5b4f28e1e29f46da7 (127.31.12.67:36935): tablet 1897fb8efdb94c35986efe2518e1f0e3 (table test-table [id=503800fd9f94409d9377bd1cda029de1]) successfully deleted
Sep 24 15:18:11 dist-test-slave-zwbb krb5kdc[1029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263091, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 15:18:11 dist-test-slave-zwbb krb5kdc[1029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263091, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:11.938211 1678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:11.931136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34792 (local address 127.31.12.126:39359)
0924 15:18:11.931301 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:11.931303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:11.931315 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:18:11.931339 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:11.931339 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:11.931350 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:11.931413 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:11.931954 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:11.932456 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:11.933222 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:11.933433 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:11.933568 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:11.935735 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:11.935736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:11.935736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:11.935743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:11.937137 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:11.937538 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:11.937539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:11.937539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:11.937594 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:11.937758 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:11.937758 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:11.937758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:11.937854 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:11.937927 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:11.938030 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 15:18:11.938092 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":100,"threads_started":1}
I20260924 15:18:11.939582 1073 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:18:11.939736 1073 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:18:11.941031 1214 tablet_service.cc:1467] Processing CreateTablet for tablet ed08176f772f4f308066cebd28be7ebd (DEFAULT_TABLE table=test-table [id=914e062863444d4e8bf16bb781331be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:11.941058 1501 tablet_service.cc:1467] Processing CreateTablet for tablet ed08176f772f4f308066cebd28be7ebd (DEFAULT_TABLE table=test-table [id=914e062863444d4e8bf16bb781331be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:11.941058 1359 tablet_service.cc:1467] Processing CreateTablet for tablet ed08176f772f4f308066cebd28be7ebd (DEFAULT_TABLE table=test-table [id=914e062863444d4e8bf16bb781331be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:11.941182 1501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed08176f772f4f308066cebd28be7ebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:11.941182 1359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed08176f772f4f308066cebd28be7ebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:11.941186 1214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed08176f772f4f308066cebd28be7ebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:11.942329 1679 tablet_bootstrap.cc:492] T ed08176f772f4f308066cebd28be7ebd P 1a280bc8a849478da774f8bae065653d: Bootstrap starting.
I20260924 15:18:11.942555 1681 tablet_bootstrap.cc:492] T ed08176f772f4f308066cebd28be7ebd P e4142cfb5fb54ed5aa8ceaf502688ae2: Bootstrap starting.
I20260924 15:18:11.942692 1679 tablet_bootstrap.cc:654] T ed08176f772f4f308066cebd28be7ebd P 1a280bc8a849478da774f8bae065653d: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:11.943020 1681 tablet_bootstrap.cc:654] T ed08176f772f4f308066cebd28be7ebd P e4142cfb5fb54ed5aa8ceaf502688ae2: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:11.943288 1679 tablet_bootstrap.cc:492] T ed08176f772f4f308066cebd28be7ebd P 1a280bc8a849478da774f8bae065653d: No bootstrap required, opened a new log
I20260924 15:18:11.943346 1680 tablet_bootstrap.cc:492] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7: Bootstrap starting.
I20260924 15:18:11.943359 1679 ts_tablet_manager.cc:1397] T ed08176f772f4f308066cebd28be7ebd P 1a280bc8a849478da774f8bae065653d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:11.943476 1679 raft_consensus.cc:348] T ed08176f772f4f308066cebd28be7ebd P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:11.943576 1679 raft_consensus.cc:374] T ed08176f772f4f308066cebd28be7ebd P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:11.943588 1681 tablet_bootstrap.cc:492] T ed08176f772f4f308066cebd28be7ebd P e4142cfb5fb54ed5aa8ceaf502688ae2: No bootstrap required, opened a new log
I20260924 15:18:11.943604 1679 raft_consensus.cc:729] T ed08176f772f4f308066cebd28be7ebd P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a280bc8a849478da774f8bae065653d, State: Initialized, Role: FOLLOWER
I20260924 15:18:11.943622 1681 ts_tablet_manager.cc:1397] T ed08176f772f4f308066cebd28be7ebd P e4142cfb5fb54ed5aa8ceaf502688ae2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:11.943655 1679 consensus_queue.cc:260] T ed08176f772f4f308066cebd28be7ebd P 1a280bc8a849478da774f8bae065653d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:11.943760 1679 ts_tablet_manager.cc:1428] T ed08176f772f4f308066cebd28be7ebd P 1a280bc8a849478da774f8bae065653d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:11.943756 1681 raft_consensus.cc:348] T ed08176f772f4f308066cebd28be7ebd P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:11.943836 1681 raft_consensus.cc:374] T ed08176f772f4f308066cebd28be7ebd P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:11.943866 1681 raft_consensus.cc:729] T ed08176f772f4f308066cebd28be7ebd P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4142cfb5fb54ed5aa8ceaf502688ae2, State: Initialized, Role: FOLLOWER
I20260924 15:18:11.943921 1681 consensus_queue.cc:260] T ed08176f772f4f308066cebd28be7ebd P e4142cfb5fb54ed5aa8ceaf502688ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:11.944032 1681 ts_tablet_manager.cc:1428] T ed08176f772f4f308066cebd28be7ebd P e4142cfb5fb54ed5aa8ceaf502688ae2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:11.944525 1680 tablet_bootstrap.cc:654] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:11.945248 1680 tablet_bootstrap.cc:492] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7: No bootstrap required, opened a new log
I20260924 15:18:11.945313 1680 ts_tablet_manager.cc:1397] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:18:11.945415 1680 raft_consensus.cc:348] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:11.945547 1680 raft_consensus.cc:374] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:11.945569 1680 raft_consensus.cc:729] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa584d948dc949f5b4f28e1e29f46da7, State: Initialized, Role: FOLLOWER
I20260924 15:18:11.945629 1680 consensus_queue.cc:260] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:11.945779 1680 ts_tablet_manager.cc:1428] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:12.004138 1685 raft_consensus.cc:482] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:18:12.004283 1685 raft_consensus.cc:504] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:12.004642 1685 leader_election.cc:288] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1a280bc8a849478da774f8bae065653d (127.31.12.65:42361), e4142cfb5fb54ed5aa8ceaf502688ae2 (127.31.12.66:32925)
I20260924 15:18:12.008867 1689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:12.004908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:44089 (local address 127.31.12.66:32925)
0924 15:18:12.005065 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:12.005066 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:12.005165 (+ 99us) server_negotiation.cc:407] Connection header received
0924 15:18:12.005269 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:12.005270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:12.005278 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:12.005331 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:12.005514 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:12.006233 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:12.007918 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:12.008501 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:12.008577 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:12.008585 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:12.008751 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 15:18:12.008790 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260924 15:18:12.008867 1687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:12.004850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:38461 (local address 127.31.12.65:42361)
0924 15:18:12.005031 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:12.005033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:12.005058 (+ 25us) server_negotiation.cc:407] Connection header received
0924 15:18:12.005117 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:12.005118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:12.005129 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:12.005195 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:12.005399 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:12.006219 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:12.007918 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:12.008495 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:12.008576 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:12.008585 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:12.008751 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 15:18:12.008793 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":100,"threads_started":1}
I20260924 15:18:12.008886 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:12.004779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:42361 (local address 127.31.12.67:38461)
0924 15:18:12.004992 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:12.005005 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:12.005054 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:12.005223 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:12.005223 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:12.005226 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:12.005299 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:12.005301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:12.006327 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:12.006329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:12.007826 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:12.007830 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:12.008614 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:12.008615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:12.008672 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:12.008674 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:18:12.008729 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 15:18:12.008781 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":130,"thread_start_us":132,"threads_started":1}
I20260924 15:18:12.008895 1688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:12.004818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:32925 (local address 127.31.12.67:44089)
0924 15:18:12.005104 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:12.005112 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:12.005167 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:12.005358 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:12.005359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:12.005361 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:12.005405 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:12.005406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:12.006327 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:12.006329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:12.007826 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:12.007830 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:12.008593 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:12.008594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:12.008662 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:12.008668 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 15:18:12.008730 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 15:18:12.008781 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":94,"thread_start_us":157,"threads_started":1}
I20260924 15:18:12.009208 1235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed08176f772f4f308066cebd28be7ebd" candidate_uuid: "fa584d948dc949f5b4f28e1e29f46da7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a280bc8a849478da774f8bae065653d" is_pre_election: true
I20260924 15:18:12.009307 1235 raft_consensus.cc:2454] T ed08176f772f4f308066cebd28be7ebd P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa584d948dc949f5b4f28e1e29f46da7 in term 0.
I20260924 15:18:12.009482 1379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed08176f772f4f308066cebd28be7ebd" candidate_uuid: "fa584d948dc949f5b4f28e1e29f46da7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" is_pre_election: true
I20260924 15:18:12.009564 1379 raft_consensus.cc:2454] T ed08176f772f4f308066cebd28be7ebd P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa584d948dc949f5b4f28e1e29f46da7 in term 0.
I20260924 15:18:12.009518 1455 leader_election.cc:302] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a280bc8a849478da774f8bae065653d, fa584d948dc949f5b4f28e1e29f46da7; no voters:
I20260924 15:18:12.009660 1685 raft_consensus.cc:2781] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:18:12.009706 1685 raft_consensus.cc:482] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:18:12.009727 1685 raft_consensus.cc:3037] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:12.010388 1685 raft_consensus.cc:504] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:12.010546 1685 leader_election.cc:288] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [CANDIDATE]: Term 1 election: Requested vote from peers 1a280bc8a849478da774f8bae065653d (127.31.12.65:42361), e4142cfb5fb54ed5aa8ceaf502688ae2 (127.31.12.66:32925)
I20260924 15:18:12.010674 1235 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed08176f772f4f308066cebd28be7ebd" candidate_uuid: "fa584d948dc949f5b4f28e1e29f46da7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1a280bc8a849478da774f8bae065653d"
I20260924 15:18:12.010677 1379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed08176f772f4f308066cebd28be7ebd" candidate_uuid: "fa584d948dc949f5b4f28e1e29f46da7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2"
I20260924 15:18:12.010751 1235 raft_consensus.cc:3037] T ed08176f772f4f308066cebd28be7ebd P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:12.010771 1379 raft_consensus.cc:3037] T ed08176f772f4f308066cebd28be7ebd P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:12.011344 1235 raft_consensus.cc:2454] T ed08176f772f4f308066cebd28be7ebd P 1a280bc8a849478da774f8bae065653d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa584d948dc949f5b4f28e1e29f46da7 in term 1.
I20260924 15:18:12.011361 1379 raft_consensus.cc:2454] T ed08176f772f4f308066cebd28be7ebd P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa584d948dc949f5b4f28e1e29f46da7 in term 1.
I20260924 15:18:12.011519 1455 leader_election.cc:302] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a280bc8a849478da774f8bae065653d, fa584d948dc949f5b4f28e1e29f46da7; no voters:
I20260924 15:18:12.011629 1685 raft_consensus.cc:2781] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:12.011807 1685 raft_consensus.cc:686] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [term 1 LEADER]: Becoming Leader. State: Replica: fa584d948dc949f5b4f28e1e29f46da7, State: Running, Role: LEADER
I20260924 15:18:12.011865 1685 consensus_queue.cc:237] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:12.012602 1073 catalog_manager.cc:5495] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa584d948dc949f5b4f28e1e29f46da7 (127.31.12.67). New cstate: current_term: 1 leader_uuid: "fa584d948dc949f5b4f28e1e29f46da7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } health_report { overall_health: HEALTHY } } }
I20260924 15:18:12.021076 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:12.018147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.31.12.67:36935)
0924 15:18:12.018298 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:12.018300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:12.018307 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:18:12.018353 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:12.018354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:12.018372 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:12.018480 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:18:12.018637 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:12.019240 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:12.020115 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:12.020365 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:12.020466 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:12.020669 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:18:12.020794 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:18:12.020884 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:12.020948 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 15:18:12.020970 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260924 15:18:12.025815 1687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:12.023784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:57219 (local address 127.31.12.65:42361)
0924 15:18:12.023838 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:12.023840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:12.023853 (+ 13us) server_negotiation.cc:407] Connection header received
0924 15:18:12.023874 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:12.023875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:12.023884 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:12.023945 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:18:12.024030 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:12.024459 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:12.025174 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:12.025353 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:12.025404 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:12.025628 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:18:12.025654 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:18:12.025695 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:12.025746 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 15:18:12.025765 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 15:18:12.025838 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:12.023592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:42361 (local address 127.31.12.67:57219)
0924 15:18:12.023765 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:12.023776 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:12.023834 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:12.023945 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:12.023946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:12.023947 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:18:12.023989 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:12.023990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:12.024517 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:12.024518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:12.025102 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:12.025103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:12.025422 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:12.025423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:12.025530 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:12.025553 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:18:12.025709 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:18:12.025710 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 15:18:12.025744 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 15:18:12.025778 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":65,"threads_started":1}
I20260924 15:18:12.027787 1379 raft_consensus.cc:1261] T ed08176f772f4f308066cebd28be7ebd P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 1 FOLLOWER]: Refusing update from remote peer fa584d948dc949f5b4f28e1e29f46da7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:12.027787 1235 raft_consensus.cc:1261] T ed08176f772f4f308066cebd28be7ebd P 1a280bc8a849478da774f8bae065653d [term 1 FOLLOWER]: Refusing update from remote peer fa584d948dc949f5b4f28e1e29f46da7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:12.028120 1685 consensus_queue.cc:1035] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:12.028240 1690 consensus_queue.cc:1035] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:12.035403 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:12.032979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:41763 (local address 127.31.12.67:36935)
0924 15:18:12.033037 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:12.033039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:12.033047 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:18:12.033113 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:12.033114 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:12.033123 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:12.033176 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:18:12.033281 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:12.033810 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:12.034691 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:12.034949 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:12.035036 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:12.035190 (+ 154us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:18:12.035220 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:18:12.035265 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:12.035319 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 15:18:12.035336 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 15:18:12.035437 1701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:12.032847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:36935 (local address 127.31.12.65:41763)
0924 15:18:12.032981 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:12.033009 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:12.033062 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:12.033180 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:12.033180 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:12.033181 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:18:12.033224 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:12.033225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:12.033891 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:12.033892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:12.034606 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:12.034609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:12.035039 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:12.035040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:12.035096 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:12.035100 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:18:12.035264 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:18:12.035265 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 15:18:12.035315 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 15:18:12.035357 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":87,"threads_started":1}
I20260924 15:18:12.044536 1075 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:18:12.044651 1075 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:18:12.045329 1075 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788: Sending DeleteTablet for 3 replicas of tablet ed08176f772f4f308066cebd28be7ebd
I20260924 15:18:12.045615 1359 tablet_service.cc:1514] Processing DeleteTablet for tablet ed08176f772f4f308066cebd28be7ebd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:12 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:56086
I20260924 15:18:12.045634 1501 tablet_service.cc:1514] Processing DeleteTablet for tablet ed08176f772f4f308066cebd28be7ebd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:12 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:35062
I20260924 15:18:12.045717 1671 tablet_replica.cc:330] T ed08176f772f4f308066cebd28be7ebd P e4142cfb5fb54ed5aa8ceaf502688ae2: stopping tablet replica
I20260924 15:18:12.045763 1671 raft_consensus.cc:2229] T ed08176f772f4f308066cebd28be7ebd P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:18:12.045797 1671 raft_consensus.cc:2258] T ed08176f772f4f308066cebd28be7ebd P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:12.045835 1214 tablet_service.cc:1514] Processing DeleteTablet for tablet ed08176f772f4f308066cebd28be7ebd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:12 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:56976
I20260924 15:18:12.045874 1668 tablet_replica.cc:330] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7: stopping tablet replica
I20260924 15:18:12.045922 1668 raft_consensus.cc:2229] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:18:12.045933 1670 tablet_replica.cc:330] T ed08176f772f4f308066cebd28be7ebd P 1a280bc8a849478da774f8bae065653d: stopping tablet replica
I20260924 15:18:12.045990 1670 raft_consensus.cc:2229] T ed08176f772f4f308066cebd28be7ebd P 1a280bc8a849478da774f8bae065653d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:18:12.045958 1075 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34792:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 15:18:12.046034 1670 raft_consensus.cc:2258] T ed08176f772f4f308066cebd28be7ebd P 1a280bc8a849478da774f8bae065653d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:12.046029 1668 raft_consensus.cc:2258] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 15:18:12.046082 1075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:18:12.046326 1670 ts_tablet_manager.cc:1905] T ed08176f772f4f308066cebd28be7ebd P 1a280bc8a849478da774f8bae065653d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:12.046717 1668 ts_tablet_manager.cc:1905] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:12.046756 1671 ts_tablet_manager.cc:1905] T ed08176f772f4f308066cebd28be7ebd P e4142cfb5fb54ed5aa8ceaf502688ae2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:12.048012 1670 ts_tablet_manager.cc:1918] T ed08176f772f4f308066cebd28be7ebd P 1a280bc8a849478da774f8bae065653d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:18:12.048060 1670 log.cc:1196] T ed08176f772f4f308066cebd28be7ebd P 1a280bc8a849478da774f8bae065653d: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-0/wal/wals/ed08176f772f4f308066cebd28be7ebd
I20260924 15:18:12.048286 1670 ts_tablet_manager.cc:1939] T ed08176f772f4f308066cebd28be7ebd P 1a280bc8a849478da774f8bae065653d: Deleting consensus metadata
I20260924 15:18:12.048285 1671 ts_tablet_manager.cc:1918] T ed08176f772f4f308066cebd28be7ebd P e4142cfb5fb54ed5aa8ceaf502688ae2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:18:12.048339 1671 log.cc:1196] T ed08176f772f4f308066cebd28be7ebd P e4142cfb5fb54ed5aa8ceaf502688ae2: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/ed08176f772f4f308066cebd28be7ebd
I20260924 15:18:12.048377 1668 ts_tablet_manager.cc:1918] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:18:12.048426 1668 log.cc:1196] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/ed08176f772f4f308066cebd28be7ebd
I20260924 15:18:12.048580 1671 ts_tablet_manager.cc:1939] T ed08176f772f4f308066cebd28be7ebd P e4142cfb5fb54ed5aa8ceaf502688ae2: Deleting consensus metadata
I20260924 15:18:12.048678 1668 ts_tablet_manager.cc:1939] T ed08176f772f4f308066cebd28be7ebd P fa584d948dc949f5b4f28e1e29f46da7: Deleting consensus metadata
I20260924 15:18:12.048785 1061 catalog_manager.cc:4841] TS 1a280bc8a849478da774f8bae065653d (127.31.12.65:42361): tablet ed08176f772f4f308066cebd28be7ebd (table test-table [id=914e062863444d4e8bf16bb781331be3]) successfully deleted
I20260924 15:18:12.049054 1060 catalog_manager.cc:4841] TS e4142cfb5fb54ed5aa8ceaf502688ae2 (127.31.12.66:32925): tablet ed08176f772f4f308066cebd28be7ebd (table test-table [id=914e062863444d4e8bf16bb781331be3]) successfully deleted
I20260924 15:18:12.049135 1061 catalog_manager.cc:4841] TS fa584d948dc949f5b4f28e1e29f46da7 (127.31.12.67:36935): tablet ed08176f772f4f308066cebd28be7ebd (table test-table [id=914e062863444d4e8bf16bb781331be3]) successfully deleted
I20260924 15:18:12.049657 1359 tablet_service.cc:1467] Processing CreateTablet for tablet 29573b959eae4b40afac2cab3cb1f59b (DEFAULT_TABLE table=test-table [id=ff5200581cb5498ab30edc2c6fb0a278]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:12.049664 1501 tablet_service.cc:1467] Processing CreateTablet for tablet 29573b959eae4b40afac2cab3cb1f59b (DEFAULT_TABLE table=test-table [id=ff5200581cb5498ab30edc2c6fb0a278]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:12.049772 1359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29573b959eae4b40afac2cab3cb1f59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:12.049790 1501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29573b959eae4b40afac2cab3cb1f59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:12.050930 1680 tablet_bootstrap.cc:492] T 29573b959eae4b40afac2cab3cb1f59b P fa584d948dc949f5b4f28e1e29f46da7: Bootstrap starting.
I20260924 15:18:12.051023 1681 tablet_bootstrap.cc:492] T 29573b959eae4b40afac2cab3cb1f59b P e4142cfb5fb54ed5aa8ceaf502688ae2: Bootstrap starting.
I20260924 15:18:12.051321 1680 tablet_bootstrap.cc:654] T 29573b959eae4b40afac2cab3cb1f59b P fa584d948dc949f5b4f28e1e29f46da7: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:12.051354 1681 tablet_bootstrap.cc:654] T 29573b959eae4b40afac2cab3cb1f59b P e4142cfb5fb54ed5aa8ceaf502688ae2: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:12.051756 1214 tablet_service.cc:1467] Processing CreateTablet for tablet 29573b959eae4b40afac2cab3cb1f59b (DEFAULT_TABLE table=test-table [id=ff5200581cb5498ab30edc2c6fb0a278]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:12.051865 1214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29573b959eae4b40afac2cab3cb1f59b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:12.051921 1681 tablet_bootstrap.cc:492] T 29573b959eae4b40afac2cab3cb1f59b P e4142cfb5fb54ed5aa8ceaf502688ae2: No bootstrap required, opened a new log
I20260924 15:18:12.051975 1681 ts_tablet_manager.cc:1397] T 29573b959eae4b40afac2cab3cb1f59b P e4142cfb5fb54ed5aa8ceaf502688ae2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:12.052093 1681 raft_consensus.cc:348] T 29573b959eae4b40afac2cab3cb1f59b P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:12.052181 1681 raft_consensus.cc:374] T 29573b959eae4b40afac2cab3cb1f59b P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:12.052208 1681 raft_consensus.cc:729] T 29573b959eae4b40afac2cab3cb1f59b P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4142cfb5fb54ed5aa8ceaf502688ae2, State: Initialized, Role: FOLLOWER
I20260924 15:18:12.052248 1681 consensus_queue.cc:260] T 29573b959eae4b40afac2cab3cb1f59b P e4142cfb5fb54ed5aa8ceaf502688ae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:12.052347 1681 ts_tablet_manager.cc:1428] T 29573b959eae4b40afac2cab3cb1f59b P e4142cfb5fb54ed5aa8ceaf502688ae2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:12.053133 1679 tablet_bootstrap.cc:492] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d: Bootstrap starting.
I20260924 15:18:12.053428 1679 tablet_bootstrap.cc:654] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:12.054004 1679 tablet_bootstrap.cc:492] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d: No bootstrap required, opened a new log
I20260924 15:18:12.054020 1680 tablet_bootstrap.cc:492] T 29573b959eae4b40afac2cab3cb1f59b P fa584d948dc949f5b4f28e1e29f46da7: No bootstrap required, opened a new log
I20260924 15:18:12.054090 1680 ts_tablet_manager.cc:1397] T 29573b959eae4b40afac2cab3cb1f59b P fa584d948dc949f5b4f28e1e29f46da7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 15:18:12.054093 1679 ts_tablet_manager.cc:1397] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 15:18:12.054210 1679 raft_consensus.cc:348] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:12.054248 1680 raft_consensus.cc:348] T 29573b959eae4b40afac2cab3cb1f59b P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:12.054296 1679 raft_consensus.cc:374] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:12.054313 1680 raft_consensus.cc:374] T 29573b959eae4b40afac2cab3cb1f59b P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:12.054327 1679 raft_consensus.cc:729] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a280bc8a849478da774f8bae065653d, State: Initialized, Role: FOLLOWER
I20260924 15:18:12.054334 1680 raft_consensus.cc:729] T 29573b959eae4b40afac2cab3cb1f59b P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa584d948dc949f5b4f28e1e29f46da7, State: Initialized, Role: FOLLOWER
I20260924 15:18:12.054369 1679 consensus_queue.cc:260] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:12.054375 1680 consensus_queue.cc:260] T 29573b959eae4b40afac2cab3cb1f59b P fa584d948dc949f5b4f28e1e29f46da7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:12.054468 1679 ts_tablet_manager.cc:1428] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 15:18:12.054486 1680 ts_tablet_manager.cc:1428] T 29573b959eae4b40afac2cab3cb1f59b P fa584d948dc949f5b4f28e1e29f46da7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:12.096915 1654 raft_consensus.cc:482] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:18:12.097025 1654 raft_consensus.cc:504] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:12.097198 1654 leader_election.cc:288] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4142cfb5fb54ed5aa8ceaf502688ae2 (127.31.12.66:32925), fa584d948dc949f5b4f28e1e29f46da7 (127.31.12.67:36935)
I20260924 15:18:12.097431 1515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29573b959eae4b40afac2cab3cb1f59b" candidate_uuid: "1a280bc8a849478da774f8bae065653d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa584d948dc949f5b4f28e1e29f46da7" is_pre_election: true
I20260924 15:18:12.097427 1379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29573b959eae4b40afac2cab3cb1f59b" candidate_uuid: "1a280bc8a849478da774f8bae065653d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" is_pre_election: true
I20260924 15:18:12.097527 1515 raft_consensus.cc:2454] T 29573b959eae4b40afac2cab3cb1f59b P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a280bc8a849478da774f8bae065653d in term 0.
I20260924 15:18:12.097579 1379 raft_consensus.cc:2454] T 29573b959eae4b40afac2cab3cb1f59b P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1a280bc8a849478da774f8bae065653d in term 0.
I20260924 15:18:12.097674 1169 leader_election.cc:302] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a280bc8a849478da774f8bae065653d, fa584d948dc949f5b4f28e1e29f46da7; no voters:
I20260924 15:18:12.097746 1654 raft_consensus.cc:2781] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:18:12.097769 1654 raft_consensus.cc:482] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:18:12.097786 1654 raft_consensus.cc:3037] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:12.098654 1654 raft_consensus.cc:504] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:12.098783 1654 leader_election.cc:288] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d [CANDIDATE]: Term 1 election: Requested vote from peers e4142cfb5fb54ed5aa8ceaf502688ae2 (127.31.12.66:32925), fa584d948dc949f5b4f28e1e29f46da7 (127.31.12.67:36935)
I20260924 15:18:12.098984 1379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29573b959eae4b40afac2cab3cb1f59b" candidate_uuid: "1a280bc8a849478da774f8bae065653d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2"
I20260924 15:18:12.098990 1515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29573b959eae4b40afac2cab3cb1f59b" candidate_uuid: "1a280bc8a849478da774f8bae065653d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa584d948dc949f5b4f28e1e29f46da7"
I20260924 15:18:12.099069 1515 raft_consensus.cc:3037] T 29573b959eae4b40afac2cab3cb1f59b P fa584d948dc949f5b4f28e1e29f46da7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:12.099073 1379 raft_consensus.cc:3037] T 29573b959eae4b40afac2cab3cb1f59b P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:12.099680 1379 raft_consensus.cc:2454] T 29573b959eae4b40afac2cab3cb1f59b P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a280bc8a849478da774f8bae065653d in term 1.
I20260924 15:18:12.099759 1515 raft_consensus.cc:2454] T 29573b959eae4b40afac2cab3cb1f59b P fa584d948dc949f5b4f28e1e29f46da7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1a280bc8a849478da774f8bae065653d in term 1.
I20260924 15:18:12.099887 1168 leader_election.cc:302] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1a280bc8a849478da774f8bae065653d, e4142cfb5fb54ed5aa8ceaf502688ae2; no voters:
I20260924 15:18:12.099978 1654 raft_consensus.cc:2781] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:12.100026 1654 raft_consensus.cc:686] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d [term 1 LEADER]: Becoming Leader. State: Replica: 1a280bc8a849478da774f8bae065653d, State: Running, Role: LEADER
I20260924 15:18:12.100073 1654 consensus_queue.cc:237] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } }
I20260924 15:18:12.100740 1075 catalog_manager.cc:5495] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d reported cstate change: term changed from 0 to 1, leader changed from <none> to 1a280bc8a849478da774f8bae065653d (127.31.12.65). New cstate: current_term: 1 leader_uuid: "1a280bc8a849478da774f8bae065653d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1a280bc8a849478da774f8bae065653d" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 } health_report { overall_health: UNKNOWN } } }
I20260924 15:18:12.115644 1687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:12.113202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57018 (local address 127.31.12.65:42361)
0924 15:18:12.113256 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:12.113258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:12.113267 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:18:12.113337 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:12.113337 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:12.113345 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:12.113385 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:18:12.113538 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:12.113995 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:12.115004 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:12.115180 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:12.115234 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:12.115397 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:18:12.115432 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:18:12.115469 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:12.115548 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 15:18:12.115572 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 15:18:12.118032 1379 raft_consensus.cc:1261] T 29573b959eae4b40afac2cab3cb1f59b P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 1 FOLLOWER]: Refusing update from remote peer 1a280bc8a849478da774f8bae065653d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:12.118129 1515 raft_consensus.cc:1261] T 29573b959eae4b40afac2cab3cb1f59b P fa584d948dc949f5b4f28e1e29f46da7 [term 1 FOLLOWER]: Refusing update from remote peer 1a280bc8a849478da774f8bae065653d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:12.118255 1595 consensus_queue.cc:1035] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4142cfb5fb54ed5aa8ceaf502688ae2" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 32925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:12.118377 1649 consensus_queue.cc:1035] T 29573b959eae4b40afac2cab3cb1f59b P 1a280bc8a849478da774f8bae065653d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa584d948dc949f5b4f28e1e29f46da7" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 36935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 15:18:12.123081 1647 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 15:18:13.643002 1075 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:18:13.643100 1075 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34792:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:18:13.643859 1075 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 39eb135056df46afa99ddce9dd14a788: Sending DeleteTablet for 3 replicas of tablet 29573b959eae4b40afac2cab3cb1f59b
I20260924 15:18:13.644184 1501 tablet_service.cc:1514] Processing DeleteTablet for tablet 29573b959eae4b40afac2cab3cb1f59b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:13 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:35062
I20260924 15:18:13.644309 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 1149
I20260924 15:18:13.644367 1359 tablet_service.cc:1514] Processing DeleteTablet for tablet 29573b959eae4b40afac2cab3cb1f59b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:13 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:56086
I20260924 15:18:13.645776 1732 tablet_replica.cc:330] T 29573b959eae4b40afac2cab3cb1f59b P fa584d948dc949f5b4f28e1e29f46da7: stopping tablet replica
I20260924 15:18:13.645862 1732 raft_consensus.cc:2229] T 29573b959eae4b40afac2cab3cb1f59b P fa584d948dc949f5b4f28e1e29f46da7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:18:13.645913 1732 raft_consensus.cc:2258] T 29573b959eae4b40afac2cab3cb1f59b P fa584d948dc949f5b4f28e1e29f46da7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:13.646011 1733 tablet_replica.cc:330] T 29573b959eae4b40afac2cab3cb1f59b P e4142cfb5fb54ed5aa8ceaf502688ae2: stopping tablet replica
I20260924 15:18:13.646095 1733 raft_consensus.cc:2229] T 29573b959eae4b40afac2cab3cb1f59b P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:18:13.646134 1733 raft_consensus.cc:2258] T 29573b959eae4b40afac2cab3cb1f59b P e4142cfb5fb54ed5aa8ceaf502688ae2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:13.646194 1732 ts_tablet_manager.cc:1905] T 29573b959eae4b40afac2cab3cb1f59b P fa584d948dc949f5b4f28e1e29f46da7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:13.646399 1733 ts_tablet_manager.cc:1905] T 29573b959eae4b40afac2cab3cb1f59b P e4142cfb5fb54ed5aa8ceaf502688ae2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:13.647874 1733 ts_tablet_manager.cc:1918] T 29573b959eae4b40afac2cab3cb1f59b P e4142cfb5fb54ed5aa8ceaf502688ae2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 15:18:13.647953 1733 log.cc:1196] T 29573b959eae4b40afac2cab3cb1f59b P e4142cfb5fb54ed5aa8ceaf502688ae2: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/29573b959eae4b40afac2cab3cb1f59b
I20260924 15:18:13.648166 1733 ts_tablet_manager.cc:1939] T 29573b959eae4b40afac2cab3cb1f59b P e4142cfb5fb54ed5aa8ceaf502688ae2: Deleting consensus metadata
I20260924 15:18:13.648228 1732 ts_tablet_manager.cc:1918] T 29573b959eae4b40afac2cab3cb1f59b P fa584d948dc949f5b4f28e1e29f46da7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 15:18:13.648293 1732 log.cc:1196] T 29573b959eae4b40afac2cab3cb1f59b P fa584d948dc949f5b4f28e1e29f46da7: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/29573b959eae4b40afac2cab3cb1f59b
I20260924 15:18:13.648490 1060 catalog_manager.cc:4841] TS e4142cfb5fb54ed5aa8ceaf502688ae2 (127.31.12.66:32925): tablet 29573b959eae4b40afac2cab3cb1f59b (table test-table [id=ff5200581cb5498ab30edc2c6fb0a278]) successfully deleted
I20260924 15:18:13.648521 1732 ts_tablet_manager.cc:1939] T 29573b959eae4b40afac2cab3cb1f59b P fa584d948dc949f5b4f28e1e29f46da7: Deleting consensus metadata
I20260924 15:18:13.648797 1061 catalog_manager.cc:4841] TS fa584d948dc949f5b4f28e1e29f46da7 (127.31.12.67:36935): tablet 29573b959eae4b40afac2cab3cb1f59b (table test-table [id=ff5200581cb5498ab30edc2c6fb0a278]) successfully deleted
W20260924 15:18:13.651706 1061 connection.cc:535] client connection to 127.31.12.65:42361 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 15:18:13.651818 1061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 15:18:13.652021 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 1293
I20260924 15:18:13.657677 1735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:13.652175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:42361 (local address 127.0.0.1:57032)
0924 15:18:13.657571 (+ 5396us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:13.657629 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.65:42361: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5364,"thread_start_us":56,"threads_started":1}
W20260924 15:18:13.657816 1061 catalog_manager.cc:4584] TS 1a280bc8a849478da774f8bae065653d (127.31.12.65:42361): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 29573b959eae4b40afac2cab3cb1f59b: Network error: Client connection negotiation failed: client connection to 127.31.12.65:42361: connect: Connection refused (error 111)
I20260924 15:18:13.657861 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 1435
I20260924 15:18:13.664121 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 1042
2026-09-24T15:18:13Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5501 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:18:13 dist-test-slave-zwbb krb5kdc[1738](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:18:13 dist-test-slave-zwbb krb5kdc[1738](info): set up 2 sockets
Sep 24 15:18:13 dist-test-slave-zwbb krb5kdc[1738](info): commencing operation
krb5kdc: starting...
W20260924 15:18:14.707830 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:18:14 dist-test-slave-zwbb krb5kdc[1738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:18:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:18:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:18:14.824358 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:44059
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:33121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:44059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:14.903709 1751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:14.903959 1751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:14.903995 1751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:14.905140 1751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:18:14.905211 1751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:14.905236 1751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:14.905257 1751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:18:14.905287 1751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:18:14.906795 1751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:44059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:44059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:14.907075 1751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:14.907299 1751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:14.912195 1757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:14.912178 1756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:14.912178 1759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:14.912626 1751 server_base.cc:1034] running on GCE node
I20260924 15:18:14.912804 1751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:14.913079 1751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:14.914234 1751 hybrid_clock.cc:648] HybridClock initialized: now 1790263094914220 us; error 28 us; skew 500 ppm
Sep 24 15:18:14 dist-test-slave-zwbb krb5kdc[1738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263094, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:14.920816 1751 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:18:14.921283 1751 webserver.cc:460] Webserver started at http://127.31.12.126:33945/ using document root <none> and password file <none>
I20260924 15:18:14.921550 1751 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:14.921597 1751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:14.921689 1751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:14.922683 1751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "0ab4e2e6bb4d4ec2861007baf543efe6"
format_stamp: "Formatted at 2026-09-24 15:18:14 on dist-test-slave-zwbb"
server_key: "8b3e0d620a6c6317c86284c46b779bfc"
server_key_iv: "c95f6f6f7623423c58f418366a29f65d"
server_key_version: "encryptionkey@0"
I20260924 15:18:14.922986 1751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "0ab4e2e6bb4d4ec2861007baf543efe6"
format_stamp: "Formatted at 2026-09-24 15:18:14 on dist-test-slave-zwbb"
server_key: "8b3e0d620a6c6317c86284c46b779bfc"
server_key_iv: "c95f6f6f7623423c58f418366a29f65d"
server_key_version: "encryptionkey@0"
I20260924 15:18:14.924268 1751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:18:14.924952 1766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:14.925122 1751 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:14.925226 1751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "0ab4e2e6bb4d4ec2861007baf543efe6"
format_stamp: "Formatted at 2026-09-24 15:18:14 on dist-test-slave-zwbb"
server_key: "8b3e0d620a6c6317c86284c46b779bfc"
server_key_iv: "c95f6f6f7623423c58f418366a29f65d"
server_key_version: "encryptionkey@0"
I20260924 15:18:14.925333 1751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:14.934605 1751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:14.936355 1751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:14.936515 1751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:14.940649 1751 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:44059
I20260924 15:18:14.940680 1818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:44059 every 8 connection(s)
I20260924 15:18:14.941032 1751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:18:14.941937 1819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:14.944195 1819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6: Bootstrap starting.
I20260924 15:18:14.944802 1819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:14.945140 1819 log.cc:824] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:14.945909 1819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6: No bootstrap required, opened a new log
I20260924 15:18:14.947546 1819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab4e2e6bb4d4ec2861007baf543efe6" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44059 } }
I20260924 15:18:14.947682 1819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:14.947705 1819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ab4e2e6bb4d4ec2861007baf543efe6, State: Initialized, Role: FOLLOWER
I20260924 15:18:14.947803 1819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab4e2e6bb4d4ec2861007baf543efe6" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44059 } }
I20260924 15:18:14.947906 1819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:18:14.947943 1819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:18:14.947996 1819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:14.948596 1819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab4e2e6bb4d4ec2861007baf543efe6" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44059 } }
I20260924 15:18:14.948628 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 1751
I20260924 15:18:14.948710 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:18:14.948722 1819 leader_election.cc:302] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ab4e2e6bb4d4ec2861007baf543efe6; no voters:
I20260924 15:18:14.948896 31793 external_mini_cluster.cc:1428] Setting key a11427482046493de248aeee415db1d6
I20260924 15:18:14.948913 1819 leader_election.cc:288] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:18:14.948998 1822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:14.949204 1819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:18:14.949375 1822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6 [term 1 LEADER]: Becoming Leader. State: Replica: 0ab4e2e6bb4d4ec2861007baf543efe6, State: Running, Role: LEADER
I20260924 15:18:14.949697 1822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab4e2e6bb4d4ec2861007baf543efe6" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44059 } }
I20260924 15:18:14.950162 1824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ab4e2e6bb4d4ec2861007baf543efe6. Latest consensus state: current_term: 1 leader_uuid: "0ab4e2e6bb4d4ec2861007baf543efe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab4e2e6bb4d4ec2861007baf543efe6" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44059 } } }
I20260924 15:18:14.950281 1824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6 [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:14.950681 1823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ab4e2e6bb4d4ec2861007baf543efe6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ab4e2e6bb4d4ec2861007baf543efe6" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44059 } } }
I20260924 15:18:14.951107 1823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6 [sys.catalog]: This master's current role is: LEADER
W20260924 15:18:14.951519 1839 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:18:14.951574 1839 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:18:14.951618 1833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:18:14.951776 1833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:18:14.953382 1833 catalog_manager.cc:1347] Generated new cluster ID: 438d6f3876ce4397a2038bc180d3dd0e
I20260924 15:18:14.953456 1833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:18:14 dist-test-slave-zwbb krb5kdc[1738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:14.959618 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:14.949375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59630 (local address 127.31.12.126:44059)
0924 15:18:14.949623 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:14.949633 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:14.949670 (+ 37us) server_negotiation.cc:407] Connection header received
0924 15:18:14.949802 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:14.949810 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:14.949869 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:14.950030 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:14.950910 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:14.952141 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:14.953203 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:14.953775 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:14.953905 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:14.956246 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:14.956250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:14.956251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:14.956262 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:14.958074 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:14.958458 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:14.958461 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:14.958461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:14.958570 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:14.958783 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:14.958784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:14.958784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:14.959021 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:14.959191 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:14.959298 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 15:18:14.959454 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":87,"threads_started":1}
I20260924 15:18:14.974112 1833 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:18:14.974651 1833 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:18:14.984910 1833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ab4e2e6bb4d4ec2861007baf543efe6: Generated new TSK 0
I20260924 15:18:14.985100 1833 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
I20260924 15:18:15.014678 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:44059
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:33121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:15.094939 1845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:15.095134 1845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:15.095157 1845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:15.096342 1845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:15.096386 1845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:15.096412 1845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:18:15.097864 1845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:44059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:15.098091 1845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:15.098340 1845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:15.103149 1853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:15.103310 1845 server_base.cc:1034] running on GCE node
W20260924 15:18:15.103163 1851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:15.103155 1850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:15.103700 1845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:15.103967 1845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:15.105132 1845 hybrid_clock.cc:648] HybridClock initialized: now 1790263095105126 us; error 36 us; skew 500 ppm
Sep 24 15:18:15 dist-test-slave-zwbb krb5kdc[1738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263095, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:15.107577 1845 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:18:15.108000 1845 webserver.cc:460] Webserver started at http://127.31.12.65:39177/ using document root <none> and password file <none>
I20260924 15:18:15.108220 1845 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:15.108289 1845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:15.108438 1845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:15.109483 1845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "261e69048fa64136950df23eac6f5e64"
format_stamp: "Formatted at 2026-09-24 15:18:15 on dist-test-slave-zwbb"
server_key: "e5cfe24882a0881f7be45d147479a094"
server_key_iv: "a103477b696d5177a9dc292259720066"
server_key_version: "encryptionkey@0"
I20260924 15:18:15.109858 1845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "261e69048fa64136950df23eac6f5e64"
format_stamp: "Formatted at 2026-09-24 15:18:15 on dist-test-slave-zwbb"
server_key: "e5cfe24882a0881f7be45d147479a094"
server_key_iv: "a103477b696d5177a9dc292259720066"
server_key_version: "encryptionkey@0"
I20260924 15:18:15.111152 1845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:15.111878 1862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:15.112057 1845 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:15.112152 1845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "261e69048fa64136950df23eac6f5e64"
format_stamp: "Formatted at 2026-09-24 15:18:15 on dist-test-slave-zwbb"
server_key: "e5cfe24882a0881f7be45d147479a094"
server_key_iv: "a103477b696d5177a9dc292259720066"
server_key_version: "encryptionkey@0"
I20260924 15:18:15.112238 1845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:15.141353 1845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:15.143226 1845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:15.143392 1845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:15.143632 1845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:15.143953 1845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:15.144007 1845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:15.144047 1845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:15.144074 1845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:15.149953 1845 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:44449
I20260924 15:18:15.149992 1975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:44449 every 8 connection(s)
I20260924 15:18:15.150358 1845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:18:15 dist-test-slave-zwbb krb5kdc[1738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263095, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:15.159394 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:15.151079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:38555 (local address 127.31.12.126:44059)
0924 15:18:15.151134 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:15.151136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:15.151345 (+ 209us) server_negotiation.cc:407] Connection header received
0924 15:18:15.151886 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:15.151888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:15.151906 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:15.151957 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:15.152865 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:15.153415 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:15.154416 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:15.154643 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:15.154719 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:15.156686 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:15.156687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:15.156688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:15.156696 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:15.158003 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:15.158412 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:15.158413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:15.158414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:15.158483 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:15.158733 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:15.158733 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:15.158734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:15.158847 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:15.158921 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:15.159216 (+ 295us) server_negotiation.cc:299] Negotiation successful
0924 15:18:15.159306 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 15:18:15.159547 1978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:15.151053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:44059 (local address 127.31.12.65:38555)
0924 15:18:15.151233 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:15.151268 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:15.151770 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:15.152036 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:15.152039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:15.152580 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:15.152760 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:15.152764 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:15.153545 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:15.153546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:15.154313 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:15.154315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:15.154727 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:15.154728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:15.154853 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:15.155284 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:15.155292 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:15.156588 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:15.158090 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:15.158096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:15.158098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:15.158306 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:15.158552 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:15.158553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:15.158553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:15.158664 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:15.158940 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:15.158947 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:15.159076 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 15:18:15.159179 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 15:18:15.159327 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260924 15:18:15.160127 1976 heartbeater.cc:347] Connected to a master server at 127.31.12.126:44059
I20260924 15:18:15.160205 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 1845
I20260924 15:18:15.160221 1976 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:15.160295 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:18:15.160427 1976 heartbeater.cc:511] Master 127.31.12.126:44059 requested a full tablet report, sending...
I20260924 15:18:15.160506 31793 external_mini_cluster.cc:1428] Setting key cfe5c862a88aa23551ce773e5e538abe
I20260924 15:18:15.160992 1783 ts_manager.cc:194] Registered new tserver with Master: 261e69048fa64136950df23eac6f5e64 (127.31.12.65:44449)
I20260924 15:18:15.161686 1783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:38555
WARNING: no policy specified for kudu/127.31.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
I20260924 15:18:15.184257 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:44059
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:33121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:15.263646 1981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:15.263842 1981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:15.263867 1981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:15.265089 1981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:15.265130 1981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:15.265183 1981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:18:15.266599 1981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:44059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:15.266832 1981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:15.267062 1981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:15.271831 1986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:15.271858 1987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:15.271905 1989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:15.271934 1981 server_base.cc:1034] running on GCE node
I20260924 15:18:15.272262 1981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:15.272521 1981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:15.273681 1981 hybrid_clock.cc:648] HybridClock initialized: now 1790263095273670 us; error 56 us; skew 500 ppm
Sep 24 15:18:15 dist-test-slave-zwbb krb5kdc[1738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263095, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:15.275991 1981 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:18:15.276507 1981 webserver.cc:460] Webserver started at http://127.31.12.66:42781/ using document root <none> and password file <none>
I20260924 15:18:15.276733 1981 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:15.276800 1981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:15.276937 1981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:15.278009 1981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "7a18a881ca8347aeb12e47b93fcfb03c"
format_stamp: "Formatted at 2026-09-24 15:18:15 on dist-test-slave-zwbb"
server_key: "2a0ac9bc39c404bdd2e27d52ab90cb10"
server_key_iv: "13590563f25118bbee780c619371a14f"
server_key_version: "encryptionkey@0"
I20260924 15:18:15.278370 1981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "7a18a881ca8347aeb12e47b93fcfb03c"
format_stamp: "Formatted at 2026-09-24 15:18:15 on dist-test-slave-zwbb"
server_key: "2a0ac9bc39c404bdd2e27d52ab90cb10"
server_key_iv: "13590563f25118bbee780c619371a14f"
server_key_version: "encryptionkey@0"
I20260924 15:18:15.279718 1981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:18:15.280493 1997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:15.280670 1981 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:15.280771 1981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "7a18a881ca8347aeb12e47b93fcfb03c"
format_stamp: "Formatted at 2026-09-24 15:18:15 on dist-test-slave-zwbb"
server_key: "2a0ac9bc39c404bdd2e27d52ab90cb10"
server_key_iv: "13590563f25118bbee780c619371a14f"
server_key_version: "encryptionkey@0"
I20260924 15:18:15.280858 1981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:15.292403 1981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:15.294294 1981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:15.294456 1981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:15.294677 1981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:15.294982 1981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:15.295038 1981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:15.295078 1981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:15.295106 1981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:15.300922 1981 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:42535
I20260924 15:18:15.300971 2112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:42535 every 8 connection(s)
I20260924 15:18:15.301267 1981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:18:15 dist-test-slave-zwbb krb5kdc[1738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263095, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:15.309022 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 1981
I20260924 15:18:15.309101 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:18:15.309475 31793 external_mini_cluster.cc:1428] Setting key 0020e39613ee2e97f8c8577881bae13a
I20260924 15:18:15.310592 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:15.301999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:33667 (local address 127.31.12.126:44059)
0924 15:18:15.302067 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:15.302069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:15.302293 (+ 224us) server_negotiation.cc:407] Connection header received
0924 15:18:15.302812 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:15.302813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:15.302826 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:15.302889 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:15.303808 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:15.304313 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:15.305302 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:15.305570 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:15.305647 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:15.307642 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:15.307643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:15.307643 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:15.307651 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:15.308898 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:15.309373 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:15.309374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:15.309375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:15.309490 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:15.309826 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:15.309827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:15.309827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:15.309954 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:15.310017 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:15.310393 (+ 376us) server_negotiation.cc:299] Negotiation successful
0924 15:18:15.310497 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 15:18:15.310743 2115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:15.301974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:44059 (local address 127.31.12.66:33667)
0924 15:18:15.302151 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:15.302218 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:15.302705 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:15.302977 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:15.302983 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:15.303526 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:15.303706 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:15.303710 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:15.304409 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:15.304410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:15.305211 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:15.305212 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:15.305652 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:15.305653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:15.305777 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:15.306206 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:15.306214 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:15.307538 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:15.309007 (+ 1469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:15.309015 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:15.309017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:15.309313 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:15.309544 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:15.309545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:15.309545 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:15.309697 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:15.310078 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:15.310087 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:15.310251 (+ 164us) client_negotiation.cc:769] Sending connection context
0924 15:18:15.310358 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 15:18:15.310514 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":92,"threads_started":1}
I20260924 15:18:15.311174 2113 heartbeater.cc:347] Connected to a master server at 127.31.12.126:44059
I20260924 15:18:15.311264 2113 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:15.311458 2113 heartbeater.cc:511] Master 127.31.12.126:44059 requested a full tablet report, sending...
I20260924 15:18:15.311846 1783 ts_manager.cc:194] Registered new tserver with Master: 7a18a881ca8347aeb12e47b93fcfb03c (127.31.12.66:42535)
I20260924 15:18:15.312234 1783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:33667
WARNING: no policy specified for kudu/127.31.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
I20260924 15:18:15.334636 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:44059
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:33121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:15.413817 2118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:15.414004 2118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:15.414027 2118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:15.415239 2118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:15.415277 2118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:15.415303 2118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:18:15.416714 2118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:44059
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:15.416949 2118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:15.417148 2118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:15.421974 2123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:15.422080 2126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:15.422165 2118 server_base.cc:1034] running on GCE node
W20260924 15:18:15.422091 2124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:15.422500 2118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:15.422737 2118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:15.423882 2118 hybrid_clock.cc:648] HybridClock initialized: now 1790263095423862 us; error 37 us; skew 500 ppm
Sep 24 15:18:15 dist-test-slave-zwbb krb5kdc[1738](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263095, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:15.426432 2118 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:18:15.426834 2118 webserver.cc:460] Webserver started at http://127.31.12.67:33671/ using document root <none> and password file <none>
I20260924 15:18:15.427053 2118 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:15.427119 2118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:15.427269 2118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:15.428436 2118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "3866093906364179ac72a61a101e5c6e"
format_stamp: "Formatted at 2026-09-24 15:18:15 on dist-test-slave-zwbb"
server_key: "de2b41f565b525d9a9243cc769f27d87"
server_key_iv: "28f179ebd0863108ca72b90955731388"
server_key_version: "encryptionkey@0"
I20260924 15:18:15.428776 2118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "3866093906364179ac72a61a101e5c6e"
format_stamp: "Formatted at 2026-09-24 15:18:15 on dist-test-slave-zwbb"
server_key: "de2b41f565b525d9a9243cc769f27d87"
server_key_iv: "28f179ebd0863108ca72b90955731388"
server_key_version: "encryptionkey@0"
I20260924 15:18:15.430213 2118 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:18:15.430930 2133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:15.431095 2118 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:15.431198 2118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "3866093906364179ac72a61a101e5c6e"
format_stamp: "Formatted at 2026-09-24 15:18:15 on dist-test-slave-zwbb"
server_key: "de2b41f565b525d9a9243cc769f27d87"
server_key_iv: "28f179ebd0863108ca72b90955731388"
server_key_version: "encryptionkey@0"
I20260924 15:18:15.431270 2118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:15.462222 2118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:15.464564 2118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:15.464727 2118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:15.464989 2118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:15.465299 2118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:15.465353 2118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:15.465405 2118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:15.465456 2118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:15.471225 2118 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:36311
I20260924 15:18:15.471268 2249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:36311 every 8 connection(s)
I20260924 15:18:15.471597 2118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:18:15 dist-test-slave-zwbb krb5kdc[1738](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263095, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:15.479427 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 2118
I20260924 15:18:15.479530 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:18:15.479745 31793 external_mini_cluster.cc:1428] Setting key f4016bdf4f9f0ff3830e16ed43d857ad
I20260924 15:18:15.481251 1826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:15.472320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:56777 (local address 127.31.12.126:44059)
0924 15:18:15.472409 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:15.472411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:15.472594 (+ 183us) server_negotiation.cc:407] Connection header received
0924 15:18:15.473151 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:15.473152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:15.473165 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:15.473224 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:15.474260 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:15.474755 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:15.475762 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:15.475962 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:15.476047 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:15.478235 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:15.478237 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:15.478237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:15.478244 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:15.479812 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:15.480341 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:15.480358 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:15.480358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:15.480422 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:15.480707 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:15.480707 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:15.480708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:15.480853 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:15.480905 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:15.481128 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 15:18:15.481184 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:18:15.481528 2252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:15.472280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:44059 (local address 127.31.12.67:56777)
0924 15:18:15.472461 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:15.472510 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:15.473043 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:15.473298 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:15.473302 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:15.473960 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:15.474152 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:15.474157 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:15.474855 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:15.474856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:15.475670 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:15.475673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:15.476056 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:15.476058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:15.476187 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:15.476728 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:15.476739 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:15.478150 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:15.479918 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:15.479926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:15.479928 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:15.480262 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:15.480500 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:15.480500 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:15.480501 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:15.480638 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:15.480911 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:15.480915 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:15.481032 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 15:18:15.481115 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 15:18:15.481236 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
I20260924 15:18:15.482113 2250 heartbeater.cc:347] Connected to a master server at 127.31.12.126:44059
I20260924 15:18:15.482208 2250 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:15.482400 2250 heartbeater.cc:511] Master 127.31.12.126:44059 requested a full tablet report, sending...
I20260924 15:18:15.482792 1783 ts_manager.cc:194] Registered new tserver with Master: 3866093906364179ac72a61a101e5c6e (127.31.12.67:36311)
I20260924 15:18:15.483134 1783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:56777
I20260924 15:18:15.491568 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 15:18:15.503239 1826 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:18:15.502369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59646 (local address 127.31.12.126:44059)
0924 15:18:15.502463 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:15.502465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:15.502512 (+ 47us) server_negotiation.cc:407] Connection header received
0924 15:18:15.502572 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:15.502573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:15.502582 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:15.502631 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:15.503162 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59646: BlockingRecv error: recv got EOF from 127.0.0.1:59646 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 15:18:15.504066 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 1845
I20260924 15:18:15.509111 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 1981
I20260924 15:18:15.514298 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 2118
I20260924 15:18:15.518738 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 1751
2026-09-24T15:18:15Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1853 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:18:15 dist-test-slave-zwbb krb5kdc[2262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:18:15 dist-test-slave-zwbb krb5kdc[2262](info): set up 2 sockets
Sep 24 15:18:15 dist-test-slave-zwbb krb5kdc[2262](info): commencing operation
krb5kdc: starting...
W20260924 15:18:16.555411 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:18:16 dist-test-slave-zwbb krb5kdc[2262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:18:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:18:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:18:16.672335 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:33779
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:45259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:33779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:16.755014 2276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:16.755229 2276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:16.755254 2276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:16.756451 2276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:18:16.756491 2276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:16.756506 2276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:16.756520 2276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:18:16.756531 2276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:18:16.758129 2276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:45259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:33779
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:33779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:16.758389 2276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:16.758622 2276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:16.763633 2282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:16.763620 2281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:16.763622 2284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:16.763921 2276 server_base.cc:1034] running on GCE node
I20260924 15:18:16.764191 2276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:16.764485 2276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:16.765651 2276 hybrid_clock.cc:648] HybridClock initialized: now 1790263096765628 us; error 43 us; skew 500 ppm
Sep 24 15:18:16 dist-test-slave-zwbb krb5kdc[2262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263096, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:16.768200 2276 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:18:16.768708 2276 webserver.cc:460] Webserver started at http://127.31.12.126:41701/ using document root <none> and password file <none>
I20260924 15:18:16.768970 2276 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:16.769044 2276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:16.769196 2276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:16.770344 2276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "228c299539c34b4796e4d297f66615c6"
format_stamp: "Formatted at 2026-09-24 15:18:16 on dist-test-slave-zwbb"
server_key: "83f312019a7292acb40ea7160097613f"
server_key_iv: "0caab132ca7769dc982a8c5f631e1c29"
server_key_version: "encryptionkey@0"
I20260924 15:18:16.770725 2276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "228c299539c34b4796e4d297f66615c6"
format_stamp: "Formatted at 2026-09-24 15:18:16 on dist-test-slave-zwbb"
server_key: "83f312019a7292acb40ea7160097613f"
server_key_iv: "0caab132ca7769dc982a8c5f631e1c29"
server_key_version: "encryptionkey@0"
I20260924 15:18:16.772151 2276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:18:16.772928 2291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:16.773155 2276 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:18:16.773263 2276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "228c299539c34b4796e4d297f66615c6"
format_stamp: "Formatted at 2026-09-24 15:18:16 on dist-test-slave-zwbb"
server_key: "83f312019a7292acb40ea7160097613f"
server_key_iv: "0caab132ca7769dc982a8c5f631e1c29"
server_key_version: "encryptionkey@0"
I20260924 15:18:16.773363 2276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:16.793285 2276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:16.795262 2276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:16.795431 2276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:16.799628 2276 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:33779
I20260924 15:18:16.799664 2343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:33779 every 8 connection(s)
I20260924 15:18:16.800148 2276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:18:16.800791 2344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:16.803014 2344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6: Bootstrap starting.
I20260924 15:18:16.803540 2344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:16.803779 2344 log.cc:824] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:16.804402 2344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6: No bootstrap required, opened a new log
I20260924 15:18:16.805959 2344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228c299539c34b4796e4d297f66615c6" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33779 } }
I20260924 15:18:16.806114 2344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:16.806138 2344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 228c299539c34b4796e4d297f66615c6, State: Initialized, Role: FOLLOWER
I20260924 15:18:16.806236 2344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228c299539c34b4796e4d297f66615c6" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33779 } }
I20260924 15:18:16.806324 2344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:18:16.806360 2344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:18:16.806412 2344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:16.807246 2344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228c299539c34b4796e4d297f66615c6" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33779 } }
I20260924 15:18:16.807397 2344 leader_election.cc:302] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 228c299539c34b4796e4d297f66615c6; no voters:
I20260924 15:18:16.807487 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 2276
I20260924 15:18:16.807571 2344 leader_election.cc:288] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:18:16.807587 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:18:16.807629 2347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:16.807775 31793 external_mini_cluster.cc:1428] Setting key a9d9382bb058b8869e248d3c2abd4b15
I20260924 15:18:16.807826 2344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:18:16.807811 2347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6 [term 1 LEADER]: Becoming Leader. State: Replica: 228c299539c34b4796e4d297f66615c6, State: Running, Role: LEADER
I20260924 15:18:16.807991 2347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228c299539c34b4796e4d297f66615c6" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33779 } }
I20260924 15:18:16.808349 2348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "228c299539c34b4796e4d297f66615c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228c299539c34b4796e4d297f66615c6" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33779 } } }
I20260924 15:18:16.808331 2347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 228c299539c34b4796e4d297f66615c6. Latest consensus state: current_term: 1 leader_uuid: "228c299539c34b4796e4d297f66615c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "228c299539c34b4796e4d297f66615c6" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33779 } } }
I20260924 15:18:16.808504 2347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6 [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:16.808686 2348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6 [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:16.808863 2359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:18:16.809573 2359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:18:16.811161 2359 catalog_manager.cc:1347] Generated new cluster ID: 23fff1b96ac44b95bc211c391c7d036e
I20260924 15:18:16.811214 2359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:18:16 dist-test-slave-zwbb krb5kdc[2262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:16.819229 2356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:16.808379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33200 (local address 127.31.12.126:33779)
0924 15:18:16.808602 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:16.808613 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:16.808949 (+ 336us) server_negotiation.cc:407] Connection header received
0924 15:18:16.809378 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:16.809386 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:16.809491 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:16.809619 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:16.810624 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:16.811772 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:16.812541 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:16.812973 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:16.813093 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:16.815392 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:16.815395 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:16.815396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:16.815414 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:16.817643 (+ 2229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:16.818029 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:16.818032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:16.818032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:16.818127 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:16.818344 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:16.818345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:16.818345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:16.818602 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:16.818777 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:16.818884 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 15:18:16.819030 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":92,"threads_started":1}
I20260924 15:18:16.820140 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:18:16.838990 2356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:16.834787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.31.12.126:33779)
0924 15:18:16.834832 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:16.834834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:16.834943 (+ 109us) server_negotiation.cc:407] Connection header received
0924 15:18:16.834999 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:16.835000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:16.835012 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:16.835088 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:16.835476 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:16.836272 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:16.837279 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:16.837646 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:16.837719 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:16.838452 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:16.838454 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:16.838454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:18:16.838719 (+ 265us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:16.838738 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:18:16.838746 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:16.838811 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:16.838875 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 15:18:16.838901 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:18:16.847507 2359 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:18:16.848248 2359 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:18:16.867777 2359 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 228c299539c34b4796e4d297f66615c6: Generated new TSK 0
I20260924 15:18:16.867995 2359 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:18:16.878692 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 2276
2026-09-24T15:18:16Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1358 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:18:16 dist-test-slave-zwbb krb5kdc[2378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:18:16 dist-test-slave-zwbb krb5kdc[2378](info): set up 2 sockets
Sep 24 15:18:16 dist-test-slave-zwbb krb5kdc[2378](info): commencing operation
krb5kdc: starting...
W20260924 15:18:17.923197 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:18:18 dist-test-slave-zwbb krb5kdc[2378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:18:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:18:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:18:18.040164 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:38907
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:43431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:38907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:18.119546 2391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:18.119782 2391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:18.119822 2391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:18.120955 2391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:18:18.121023 2391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:18.121050 2391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:18.121081 2391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:18:18.121105 2391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:18:18.122571 2391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:43431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:38907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:38907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:18.122836 2391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:18.123057 2391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:18:18.127867 2391 server_base.cc:1034] running on GCE node
W20260924 15:18:18.127838 2396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:18.128010 2400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:18.127851 2397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:18.128466 2391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:18.128720 2391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:18.129882 2391 hybrid_clock.cc:648] HybridClock initialized: now 1790263098129860 us; error 34 us; skew 500 ppm
Sep 24 15:18:18 dist-test-slave-zwbb krb5kdc[2378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263098, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:18.132253 2391 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:18:18.132666 2391 webserver.cc:460] Webserver started at http://127.31.12.126:41559/ using document root <none> and password file <none>
I20260924 15:18:18.132887 2391 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:18.132954 2391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:18.133109 2391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:18.134205 2391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "85864076dbf84907aeae82e475f6f5ee"
format_stamp: "Formatted at 2026-09-24 15:18:18 on dist-test-slave-zwbb"
server_key: "603ba75bf15eafafc44012b915996f8d"
server_key_iv: "284b6eb38d55615f9494a3ed99f9293c"
server_key_version: "encryptionkey@0"
I20260924 15:18:18.134583 2391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "85864076dbf84907aeae82e475f6f5ee"
format_stamp: "Formatted at 2026-09-24 15:18:18 on dist-test-slave-zwbb"
server_key: "603ba75bf15eafafc44012b915996f8d"
server_key_iv: "284b6eb38d55615f9494a3ed99f9293c"
server_key_version: "encryptionkey@0"
I20260924 15:18:18.135897 2391 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:18:18.136633 2407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:18.136819 2391 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:18:18.136924 2391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "85864076dbf84907aeae82e475f6f5ee"
format_stamp: "Formatted at 2026-09-24 15:18:18 on dist-test-slave-zwbb"
server_key: "603ba75bf15eafafc44012b915996f8d"
server_key_iv: "284b6eb38d55615f9494a3ed99f9293c"
server_key_version: "encryptionkey@0"
I20260924 15:18:18.136998 2391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:18.146636 2391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:18.148336 2391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:18.148492 2391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:18.152427 2391 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:38907
I20260924 15:18:18.152458 2459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:38907 every 8 connection(s)
I20260924 15:18:18.152804 2391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:18:18.153836 2460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:18.154883 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 2391
I20260924 15:18:18.154999 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:18:18.155180 31793 external_mini_cluster.cc:1428] Setting key 4a118d71db748585ee6a38933fb345a7
I20260924 15:18:18.156842 2460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee: Bootstrap starting.
I20260924 15:18:18.157553 2460 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:18.157879 2460 log.cc:824] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:18.158622 2460 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee: No bootstrap required, opened a new log
I20260924 15:18:18.160908 2460 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85864076dbf84907aeae82e475f6f5ee" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38907 } }
I20260924 15:18:18.161052 2460 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:18.161075 2460 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85864076dbf84907aeae82e475f6f5ee, State: Initialized, Role: FOLLOWER
I20260924 15:18:18.161242 2460 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85864076dbf84907aeae82e475f6f5ee" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38907 } }
I20260924 15:18:18.161322 2460 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:18:18.161345 2460 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:18:18.161415 2460 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:18.161980 2460 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85864076dbf84907aeae82e475f6f5ee" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38907 } }
I20260924 15:18:18.162111 2460 leader_election.cc:302] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85864076dbf84907aeae82e475f6f5ee; no voters:
Sep 24 15:18:18 dist-test-slave-zwbb krb5kdc[2378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:18.162418 2460 leader_election.cc:288] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:18:18.162453 2465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:18.162655 2465 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee [term 1 LEADER]: Becoming Leader. State: Replica: 85864076dbf84907aeae82e475f6f5ee, State: Running, Role: LEADER
I20260924 15:18:18.162725 2460 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:18:18.162824 2465 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85864076dbf84907aeae82e475f6f5ee" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38907 } }
I20260924 15:18:18.163152 2465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85864076dbf84907aeae82e475f6f5ee. Latest consensus state: current_term: 1 leader_uuid: "85864076dbf84907aeae82e475f6f5ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85864076dbf84907aeae82e475f6f5ee" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38907 } } }
I20260924 15:18:18.163166 2467 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85864076dbf84907aeae82e475f6f5ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85864076dbf84907aeae82e475f6f5ee" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38907 } } }
I20260924 15:18:18.163395 2467 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:18.163314 2465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee [sys.catalog]: This master's current role is: LEADER
W20260924 15:18:18.164721 2480 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:18:18.164773 2480 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:18:18.164819 2474 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:18:18.165004 2474 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:18:18.166819 2474 catalog_manager.cc:1347] Generated new cluster ID: 73e682b5fa0c4a3ca88bc117bcf9d56e
I20260924 15:18:18.166870 2474 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:18:18.167066 2463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:18.155696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50290 (local address 127.31.12.126:38907)
0924 15:18:18.155878 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:18.155887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:18.155921 (+ 34us) server_negotiation.cc:407] Connection header received
0924 15:18:18.156587 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:18.156598 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:18.156661 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:18.156818 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:18.157484 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:18.158837 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:18.159576 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:18.160205 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:18.160343 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:18.163093 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:18.163098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:18.163098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:18.163110 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:18.165258 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:18.165760 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:18.165764 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:18.165764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:18.165872 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:18.166134 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:18.166135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:18.166135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:18.166436 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:18.166603 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:18.166734 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 15:18:18.166875 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":74,"threads_started":1}
I20260924 15:18:18.177824 2474 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:18:18.178545 2474 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:18:18.184830 2474 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85864076dbf84907aeae82e475f6f5ee: Generated new TSK 0
I20260924 15:18:18.185038 2474 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
I20260924 15:18:18.218189 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:38907
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:43431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:18.299283 2486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:18.299508 2486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:18.299546 2486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:18.300684 2486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:18.300745 2486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:18.300787 2486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:18:18.302256 2486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:43431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:38907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:18.302490 2486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:18.302666 2486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:18.307574 2494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:18.307548 2491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:18.307560 2492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:18.307755 2486 server_base.cc:1034] running on GCE node
I20260924 15:18:18.308037 2486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:18.308306 2486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:18.309492 2486 hybrid_clock.cc:648] HybridClock initialized: now 1790263098309480 us; error 30 us; skew 500 ppm
Sep 24 15:18:18 dist-test-slave-zwbb krb5kdc[2378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263098, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:18.311939 2486 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:18:18.312371 2486 webserver.cc:460] Webserver started at http://127.31.12.65:33015/ using document root <none> and password file <none>
I20260924 15:18:18.312606 2486 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:18.312651 2486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:18.312737 2486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:18.313750 2486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "af304be3d2274323968839f6c7dd8f14"
format_stamp: "Formatted at 2026-09-24 15:18:18 on dist-test-slave-zwbb"
server_key: "37216aa0e6852061e56e4dbd3a0d3ce2"
server_key_iv: "359057ba249e92beb4114dec46cecd02"
server_key_version: "encryptionkey@0"
I20260924 15:18:18.314050 2486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "af304be3d2274323968839f6c7dd8f14"
format_stamp: "Formatted at 2026-09-24 15:18:18 on dist-test-slave-zwbb"
server_key: "37216aa0e6852061e56e4dbd3a0d3ce2"
server_key_iv: "359057ba249e92beb4114dec46cecd02"
server_key_version: "encryptionkey@0"
I20260924 15:18:18.315253 2486 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:18:18.315891 2502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:18.316054 2486 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:18.316159 2486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "af304be3d2274323968839f6c7dd8f14"
format_stamp: "Formatted at 2026-09-24 15:18:18 on dist-test-slave-zwbb"
server_key: "37216aa0e6852061e56e4dbd3a0d3ce2"
server_key_iv: "359057ba249e92beb4114dec46cecd02"
server_key_version: "encryptionkey@0"
I20260924 15:18:18.316233 2486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:18.329939 2486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:18.331655 2486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:18.331813 2486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:18.332034 2486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:18.332341 2486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:18.332401 2486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:18.332437 2486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:18.332468 2486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:18.338220 2486 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:38127
I20260924 15:18:18.338264 2616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:38127 every 8 connection(s)
I20260924 15:18:18.338559 2486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:18:18.343056 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 2486
I20260924 15:18:18.343163 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:18:18.343369 31793 external_mini_cluster.cc:1428] Setting key 1d0b408accaf0a4bcf446797102716c8
Sep 24 15:18:18 dist-test-slave-zwbb krb5kdc[2378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263098, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:18.348572 2463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:18.339268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:46697 (local address 127.31.12.126:38907)
0924 15:18:18.339340 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:18.339343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:18.339553 (+ 210us) server_negotiation.cc:407] Connection header received
0924 15:18:18.340132 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:18.340133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:18.340143 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:18.340196 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:18.341220 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:18.341776 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:18.342769 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:18.343129 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:18.343220 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:18.345623 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:18.345624 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:18.345625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:18.345640 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:18.347069 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:18.347573 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:18.347573 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:18.347574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:18.347626 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:18.347918 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:18.347919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:18.347919 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:18.348043 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:18.348101 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:18.348443 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 15:18:18.348507 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:18:18.348913 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:18.339230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38907 (local address 127.31.12.65:46697)
0924 15:18:18.339426 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:18.339464 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:18.340040 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:18.340275 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:18.340279 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:18.340933 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:18.341140 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:18.341145 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:18.341864 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:18.341864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:18.342677 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:18.342680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:18.343274 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:18.343275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:18.343386 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:18.344019 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:18.344030 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:18.345520 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:18.347156 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:18.347163 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:18.347166 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:18.347490 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:18.347692 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:18.347693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:18.347693 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:18.347846 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:18.348120 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:18.348128 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:18.348297 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 15:18:18.348422 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 15:18:18.348618 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":103,"threads_started":1}
I20260924 15:18:18.349530 2617 heartbeater.cc:347] Connected to a master server at 127.31.12.126:38907
I20260924 15:18:18.349650 2617 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:18.349865 2617 heartbeater.cc:511] Master 127.31.12.126:38907 requested a full tablet report, sending...
I20260924 15:18:18.350456 2424 ts_manager.cc:194] Registered new tserver with Master: af304be3d2274323968839f6c7dd8f14 (127.31.12.65:38127)
I20260924 15:18:18.351104 2424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:46697
WARNING: no policy specified for kudu/127.31.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
I20260924 15:18:18.368682 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:38907
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:43431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:18.449261 2622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:18.449492 2622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:18.449563 2622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:18.450711 2622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:18.450750 2622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:18.450801 2622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:18:18.452235 2622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:43431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:38907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:18.452481 2622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:18.452718 2622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:18.457436 2628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:18.457607 2622 server_base.cc:1034] running on GCE node
W20260924 15:18:18.457479 2630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:18.457459 2627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:18.457888 2622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:18.458112 2622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:18.459260 2622 hybrid_clock.cc:648] HybridClock initialized: now 1790263098459249 us; error 27 us; skew 500 ppm
Sep 24 15:18:18 dist-test-slave-zwbb krb5kdc[2378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263098, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:18.461665 2622 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:18:18.462101 2622 webserver.cc:460] Webserver started at http://127.31.12.66:37687/ using document root <none> and password file <none>
I20260924 15:18:18.462342 2622 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:18.462406 2622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:18.462551 2622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:18.463505 2622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "c81060c08d4f412787fb16c8eebfda16"
format_stamp: "Formatted at 2026-09-24 15:18:18 on dist-test-slave-zwbb"
server_key: "3f4c23e748f90162d4ce1c85a50eca97"
server_key_iv: "46989ceb15a031ba8b4407a8f98591b8"
server_key_version: "encryptionkey@0"
I20260924 15:18:18.463876 2622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "c81060c08d4f412787fb16c8eebfda16"
format_stamp: "Formatted at 2026-09-24 15:18:18 on dist-test-slave-zwbb"
server_key: "3f4c23e748f90162d4ce1c85a50eca97"
server_key_iv: "46989ceb15a031ba8b4407a8f98591b8"
server_key_version: "encryptionkey@0"
I20260924 15:18:18.465142 2622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:18:18.465900 2637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:18.466066 2622 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:18.466174 2622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "c81060c08d4f412787fb16c8eebfda16"
format_stamp: "Formatted at 2026-09-24 15:18:18 on dist-test-slave-zwbb"
server_key: "3f4c23e748f90162d4ce1c85a50eca97"
server_key_iv: "46989ceb15a031ba8b4407a8f98591b8"
server_key_version: "encryptionkey@0"
I20260924 15:18:18.466248 2622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:18.492215 2622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:18.494514 2622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:18.494665 2622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:18.494890 2622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:18.495177 2622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:18.495226 2622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:18.495268 2622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:18.495294 2622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:18.501029 2622 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:44535
I20260924 15:18:18.501084 2752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:44535 every 8 connection(s)
I20260924 15:18:18.501365 2622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:18:18.503264 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 2622
I20260924 15:18:18.503417 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:18:18.503629 31793 external_mini_cluster.cc:1428] Setting key 156609cd62d32b48fee436af8f24e0bd
Sep 24 15:18:18 dist-test-slave-zwbb krb5kdc[2378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263098, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:18.511987 2463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:18.502106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:60867 (local address 127.31.12.126:38907)
0924 15:18:18.502198 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:18.502200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:18.502368 (+ 168us) server_negotiation.cc:407] Connection header received
0924 15:18:18.502818 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:18.502819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:18.502829 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:18.502892 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:18.503941 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:18.504718 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:18.505874 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:18.506094 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:18.506158 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:18.508633 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:18.508636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:18.508636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:18.508644 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:18.510552 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:18.510973 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:18.510975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:18.510975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:18.511055 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:18.511283 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:18.511284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:18.511284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:18.511469 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:18.511552 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:18.511779 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 15:18:18.511876 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 15:18:18.511998 2755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:18.502069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38907 (local address 127.31.12.66:60867)
0924 15:18:18.502212 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:18.502251 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:18.502738 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:18.502950 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:18.502953 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:18.503556 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:18.503780 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:18.503785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:18.504812 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:18.504813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:18.505764 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:18.505768 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:18.506181 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:18.506182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:18.506334 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:18.506845 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:18.506853 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:18.508534 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:18.510665 (+ 2131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:18.510671 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:18.510673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:18.510880 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:18.511122 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:18.511123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:18.511123 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:18.511214 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:18.511538 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:18.511542 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:18.511644 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 15:18:18.511721 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 15:18:18.511830 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260924 15:18:18.512596 2753 heartbeater.cc:347] Connected to a master server at 127.31.12.126:38907
I20260924 15:18:18.512696 2753 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:18.512883 2753 heartbeater.cc:511] Master 127.31.12.126:38907 requested a full tablet report, sending...
I20260924 15:18:18.513260 2424 ts_manager.cc:194] Registered new tserver with Master: c81060c08d4f412787fb16c8eebfda16 (127.31.12.66:44535)
WARNING: no policy specified for kudu/127.31.12.67@KRBTEST.COM; defaulting to no policy
I20260924 15:18:18.513710 2424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:60867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
I20260924 15:18:18.532002 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:38907
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:43431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:18.610595 2758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:18.610831 2758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:18.610872 2758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:18.612020 2758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:18.612087 2758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:18.612129 2758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:18:18.613554 2758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:43431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:38907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:18.613804 2758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:18.614040 2758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:18.618919 2764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:18.618973 2758 server_base.cc:1034] running on GCE node
W20260924 15:18:18.618937 2763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:18.618919 2766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:18.619374 2758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:18.619588 2758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:18.620750 2758 hybrid_clock.cc:648] HybridClock initialized: now 1790263098620738 us; error 46 us; skew 500 ppm
Sep 24 15:18:18 dist-test-slave-zwbb krb5kdc[2378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263098, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:18.623180 2758 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:18:18.623595 2758 webserver.cc:460] Webserver started at http://127.31.12.67:46587/ using document root <none> and password file <none>
I20260924 15:18:18.623813 2758 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:18.623881 2758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:18.624029 2758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:18.625056 2758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "bb0d55cff3574425ba45a048c3e2d3b8"
format_stamp: "Formatted at 2026-09-24 15:18:18 on dist-test-slave-zwbb"
server_key: "b0b50d92fd864b8b99fd8ec1aa55a7cb"
server_key_iv: "d10729d5d550d19ab3e4300b1dedf447"
server_key_version: "encryptionkey@0"
I20260924 15:18:18.625403 2758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "bb0d55cff3574425ba45a048c3e2d3b8"
format_stamp: "Formatted at 2026-09-24 15:18:18 on dist-test-slave-zwbb"
server_key: "b0b50d92fd864b8b99fd8ec1aa55a7cb"
server_key_iv: "d10729d5d550d19ab3e4300b1dedf447"
server_key_version: "encryptionkey@0"
I20260924 15:18:18.626822 2758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:18.627508 2773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:18.627733 2758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:18.627835 2758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "bb0d55cff3574425ba45a048c3e2d3b8"
format_stamp: "Formatted at 2026-09-24 15:18:18 on dist-test-slave-zwbb"
server_key: "b0b50d92fd864b8b99fd8ec1aa55a7cb"
server_key_iv: "d10729d5d550d19ab3e4300b1dedf447"
server_key_version: "encryptionkey@0"
I20260924 15:18:18.627905 2758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:18.637951 2758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:18.640214 2758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:18.640377 2758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:18.640605 2758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:18.640911 2758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:18.640969 2758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:18.641008 2758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:18.641036 2758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:18.646781 2758 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:43807
I20260924 15:18:18.646853 2888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:43807 every 8 connection(s)
I20260924 15:18:18.647163 2758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:18:18 dist-test-slave-zwbb krb5kdc[2378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263098, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:18.655645 2463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:18.647891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:37097 (local address 127.31.12.126:38907)
0924 15:18:18.647961 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:18.647963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:18.648145 (+ 182us) server_negotiation.cc:407] Connection header received
0924 15:18:18.648605 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:18.648606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:18.648616 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:18.648664 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:18.649577 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:18.650064 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:18.650987 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:18.651171 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:18.651218 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:18.653198 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:18.653200 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:18.653200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:18.653206 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:18.654541 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:18.654887 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:18.654888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:18.654888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:18.654938 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:18.655146 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:18.655147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:18.655147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:18.655246 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:18.655295 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:18.655522 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 15:18:18.655582 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 15:18:18.655853 2891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:18.647870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38907 (local address 127.31.12.67:37097)
0924 15:18:18.648027 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:18.648061 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:18.648526 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:18.648724 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:18.648727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:18.649253 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:18.649486 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:18.649493 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:18.650142 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:18.650143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:18.650915 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:18.650917 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:18.651239 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:18.651240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:18.651333 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:18.651839 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:18.651847 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:18.653119 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:18.654614 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:18.654620 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:18.654622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:18.654819 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:18.654994 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:18.654995 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:18.654995 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:18.655086 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:18.655299 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:18.655303 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:18.655405 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 15:18:18.655493 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 15:18:18.655603 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":94,"threads_started":1}
I20260924 15:18:18.656436 2889 heartbeater.cc:347] Connected to a master server at 127.31.12.126:38907
I20260924 15:18:18.656559 2889 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:18.656721 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 2758
I20260924 15:18:18.656736 2889 heartbeater.cc:511] Master 127.31.12.126:38907 requested a full tablet report, sending...
I20260924 15:18:18.656826 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:18:18.657037 31793 external_mini_cluster.cc:1428] Setting key 9a9f27b8d7ac61a1b3d7a4eb807f8de1
I20260924 15:18:18.657135 2424 ts_manager.cc:194] Registered new tserver with Master: bb0d55cff3574425ba45a048c3e2d3b8 (127.31.12.67:43807)
I20260924 15:18:18.657691 2424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:37097
I20260924 15:18:18.657997 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 15:18:18 dist-test-slave-zwbb krb5kdc[2378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263098, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 15:18:18 dist-test-slave-zwbb krb5kdc[2378](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263098, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:18.681337 2463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:18.673242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.31.12.126:38907)
0924 15:18:18.673304 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:18.673307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:18.673398 (+ 91us) server_negotiation.cc:407] Connection header received
0924 15:18:18.673504 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:18.673505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:18.673515 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:18.673581 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:18.674115 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:18.674931 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:18.675828 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:18.676038 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:18.676112 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:18.678254 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:18.678255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:18.678256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:18.678262 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:18.679993 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:18.680447 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:18.680448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:18.680448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:18.680508 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:18.680733 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:18.680734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:18.680734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:18.680940 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:18.681056 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:18.681161 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 15:18:18.681249 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 15:18:18.681890 2423 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:50300
I20260924 15:18:18.682384 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 2486
I20260924 15:18:18.689040 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 2622
I20260924 15:18:18.693429 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 2758
I20260924 15:18:18.697732 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 2391
2026-09-24T15:18:18Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1820 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:18:18 dist-test-slave-zwbb krb5kdc[2901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:18:18 dist-test-slave-zwbb krb5kdc[2901](info): set up 2 sockets
Sep 24 15:18:18 dist-test-slave-zwbb krb5kdc[2901](info): commencing operation
krb5kdc: starting...
W20260924 15:18:19.743191 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:18:19 dist-test-slave-zwbb krb5kdc[2901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:18:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:18:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:18:19.859815 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:36707
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:36707
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:19.939831 2914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:19.940029 2914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:19.940052 2914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:19.941218 2914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:18:19.941260 2914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:19.941274 2914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:19.941287 2914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:18:19.941299 2914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:18:19.942838 2914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:36707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:36707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:19.943082 2914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:19.943318 2914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:19.948328 2920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:19.948341 2919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:19.948369 2914 server_base.cc:1034] running on GCE node
W20260924 15:18:19.948359 2922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:19.948792 2914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:19.949132 2914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:19.950315 2914 hybrid_clock.cc:648] HybridClock initialized: now 1790263099950295 us; error 37 us; skew 500 ppm
Sep 24 15:18:19 dist-test-slave-zwbb krb5kdc[2901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263099, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:19.957568 2914 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:18:19.957989 2914 webserver.cc:460] Webserver started at http://127.31.12.126:34511/ using document root <none> and password file <none>
I20260924 15:18:19.958235 2914 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:19.958281 2914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:19.958419 2914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:19.959520 2914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "9335bdac9c6e4a1cbdad4aef284b03e4"
format_stamp: "Formatted at 2026-09-24 15:18:19 on dist-test-slave-zwbb"
server_key: "4ee292eeefa447f1fa9fc93db7aaa901"
server_key_iv: "e0850ef8f1271226f7a207eda7c97092"
server_key_version: "encryptionkey@0"
I20260924 15:18:19.959894 2914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "9335bdac9c6e4a1cbdad4aef284b03e4"
format_stamp: "Formatted at 2026-09-24 15:18:19 on dist-test-slave-zwbb"
server_key: "4ee292eeefa447f1fa9fc93db7aaa901"
server_key_iv: "e0850ef8f1271226f7a207eda7c97092"
server_key_version: "encryptionkey@0"
I20260924 15:18:19.961270 2914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:18:19.962109 2929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:19.962297 2914 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:19.962399 2914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "9335bdac9c6e4a1cbdad4aef284b03e4"
format_stamp: "Formatted at 2026-09-24 15:18:19 on dist-test-slave-zwbb"
server_key: "4ee292eeefa447f1fa9fc93db7aaa901"
server_key_iv: "e0850ef8f1271226f7a207eda7c97092"
server_key_version: "encryptionkey@0"
I20260924 15:18:19.962481 2914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:19.968845 2914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:19.970631 2914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:19.970790 2914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:19.974866 2914 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:36707
I20260924 15:18:19.974918 2982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:36707 every 8 connection(s)
I20260924 15:18:19.975440 2914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:18:19.976125 2983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:19.978572 2983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4: Bootstrap starting.
I20260924 15:18:19.979132 2983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:19.979390 2983 log.cc:824] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:19.980054 2983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4: No bootstrap required, opened a new log
I20260924 15:18:19.981660 2983 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9335bdac9c6e4a1cbdad4aef284b03e4" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 36707 } }
I20260924 15:18:19.981796 2983 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:19.981817 2983 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9335bdac9c6e4a1cbdad4aef284b03e4, State: Initialized, Role: FOLLOWER
I20260924 15:18:19.981961 2983 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9335bdac9c6e4a1cbdad4aef284b03e4" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 36707 } }
I20260924 15:18:19.982038 2983 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:18:19.982064 2983 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:18:19.982092 2983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:19.982920 2983 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9335bdac9c6e4a1cbdad4aef284b03e4" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 36707 } }
I20260924 15:18:19.983007 2983 leader_election.cc:302] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9335bdac9c6e4a1cbdad4aef284b03e4; no voters:
I20260924 15:18:19.983140 2983 leader_election.cc:288] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:18:19.983215 2986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:19.983397 2986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4 [term 1 LEADER]: Becoming Leader. State: Replica: 9335bdac9c6e4a1cbdad4aef284b03e4, State: Running, Role: LEADER
I20260924 15:18:19.983410 2983 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:18:19.983518 2986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9335bdac9c6e4a1cbdad4aef284b03e4" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 36707 } }
I20260924 15:18:19.983888 2988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9335bdac9c6e4a1cbdad4aef284b03e4. Latest consensus state: current_term: 1 leader_uuid: "9335bdac9c6e4a1cbdad4aef284b03e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9335bdac9c6e4a1cbdad4aef284b03e4" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 36707 } } }
I20260924 15:18:19.983876 2987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9335bdac9c6e4a1cbdad4aef284b03e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9335bdac9c6e4a1cbdad4aef284b03e4" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 36707 } } }
I20260924 15:18:19.983981 2988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4 [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:19.983989 2987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4 [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:19.984196 2995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:18:19.984483 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 2914
I20260924 15:18:19.984607 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:18:19.984757 2995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:18:19.984799 31793 external_mini_cluster.cc:1428] Setting key 64c8b8c4c58e6ddbd0b5e3179d80832b
I20260924 15:18:19.986461 2995 catalog_manager.cc:1347] Generated new cluster ID: 838b0791427d4059a2e2a7bc5d2d186d
I20260924 15:18:19.986522 2995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:18:19 dist-test-slave-zwbb krb5kdc[2901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263099, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:19.997058 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:19.985501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59114 (local address 127.31.12.126:36707)
0924 15:18:19.985956 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:19.985965 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:19.986005 (+ 40us) server_negotiation.cc:407] Connection header received
0924 15:18:19.986129 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:19.986134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:19.986221 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:19.987167 (+ 946us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:19.987272 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:19.988645 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:19.989673 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:19.990359 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:19.990497 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:19.993302 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:19.993307 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:19.993307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:19.993319 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:19.995288 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:19.995702 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:19.995705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:19.995706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:19.995811 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:19.996023 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:19.996023 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:19.996024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:19.996318 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:19.996460 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:19.996578 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 15:18:19.996807 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":65,"threads_started":1}
I20260924 15:18:20.000102 2995 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:18:20.000579 2995 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:18:20.009701 2995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9335bdac9c6e4a1cbdad4aef284b03e4: Generated new TSK 0
I20260924 15:18:20.009928 2995 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
I20260924 15:18:20.036446 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:36707
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:20.117144 3009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:20.117450 3009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:20.117482 3009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:20.118731 3009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:20.118773 3009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:20.118800 3009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:18:20.120338 3009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:36707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:20.120591 3009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:20.120846 3009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:20.125787 3014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:20.125869 3015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:20.125833 3017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:20.126027 3009 server_base.cc:1034] running on GCE node
I20260924 15:18:20.126263 3009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:20.126508 3009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:20.127664 3009 hybrid_clock.cc:648] HybridClock initialized: now 1790263100127649 us; error 31 us; skew 500 ppm
Sep 24 15:18:20 dist-test-slave-zwbb krb5kdc[2901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263100, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:20.130291 3009 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:18:20.130759 3009 webserver.cc:460] Webserver started at http://127.31.12.65:44027/ using document root <none> and password file <none>
I20260924 15:18:20.131009 3009 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:20.131076 3009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:20.131235 3009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:20.132355 3009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "f17dd4bf30a84e848a43e88737ad5053"
format_stamp: "Formatted at 2026-09-24 15:18:20 on dist-test-slave-zwbb"
server_key: "f24a75803539916522c610274cb2a97b"
server_key_iv: "2fd4b07be6d8549de07cc1a2fd187f72"
server_key_version: "encryptionkey@0"
I20260924 15:18:20.132733 3009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "f17dd4bf30a84e848a43e88737ad5053"
format_stamp: "Formatted at 2026-09-24 15:18:20 on dist-test-slave-zwbb"
server_key: "f24a75803539916522c610274cb2a97b"
server_key_iv: "2fd4b07be6d8549de07cc1a2fd187f72"
server_key_version: "encryptionkey@0"
I20260924 15:18:20.134215 3009 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:18:20.135056 3024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:20.135242 3009 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:18:20.135352 3009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "f17dd4bf30a84e848a43e88737ad5053"
format_stamp: "Formatted at 2026-09-24 15:18:20 on dist-test-slave-zwbb"
server_key: "f24a75803539916522c610274cb2a97b"
server_key_iv: "2fd4b07be6d8549de07cc1a2fd187f72"
server_key_version: "encryptionkey@0"
I20260924 15:18:20.135433 3009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:20.156030 3009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:20.158404 3009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:20.158584 3009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:20.158836 3009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:20.159173 3009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:20.159235 3009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:20.159276 3009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:20.159308 3009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:20.165426 3009 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:40457
I20260924 15:18:20.165560 3138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:40457 every 8 connection(s)
I20260924 15:18:20.165859 3009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:18:20.171835 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 3009
I20260924 15:18:20.171970 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:18:20.172149 31793 external_mini_cluster.cc:1428] Setting key d8605faa1f13bb4f08ec3a0d66988351
Sep 24 15:18:20 dist-test-slave-zwbb krb5kdc[2901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263100, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:20.176427 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:20.166662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:46547 (local address 127.31.12.126:36707)
0924 15:18:20.166745 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:20.166748 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:20.166936 (+ 188us) server_negotiation.cc:407] Connection header received
0924 15:18:20.167914 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:20.167915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:20.167927 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:20.167983 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:20.168598 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:20.169103 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:20.170293 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:20.170516 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:20.170580 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:20.173053 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:20.173055 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:20.173055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:20.173064 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:20.174773 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:20.175340 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:20.175342 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:20.175342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:20.175405 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:20.175727 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:20.175728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:20.175728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:20.175861 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:20.175940 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:20.176230 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 15:18:20.176335 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:18:20.176612 3141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:20.166626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:36707 (local address 127.31.12.65:46547)
0924 15:18:20.166815 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:20.166855 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:20.167813 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:20.168044 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:20.168047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:20.168315 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:20.168524 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:20.168528 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:20.169262 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:20.169263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:20.170210 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:20.170213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:20.170603 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:20.170604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:20.170711 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:20.171326 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:20.171335 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:20.172934 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:20.174857 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:20.174866 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:20.174868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:20.175241 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:20.175481 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:20.175482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:20.175483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:20.175617 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:20.175930 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:20.175937 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:20.176085 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 15:18:20.176195 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 15:18:20.176364 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":101,"threads_started":1}
I20260924 15:18:20.177181 3139 heartbeater.cc:347] Connected to a master server at 127.31.12.126:36707
I20260924 15:18:20.177277 3139 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:20.177536 3139 heartbeater.cc:511] Master 127.31.12.126:36707 requested a full tablet report, sending...
I20260924 15:18:20.178092 2946 ts_manager.cc:194] Registered new tserver with Master: f17dd4bf30a84e848a43e88737ad5053 (127.31.12.65:40457)
I20260924 15:18:20.178712 2946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:46547
WARNING: no policy specified for kudu/127.31.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
I20260924 15:18:20.202018 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:36707
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:20.282670 3144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:20.282886 3144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:20.282910 3144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:20.284245 3144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:20.284309 3144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:20.284355 3144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:18:20.285948 3144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:36707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:20.286198 3144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:20.286435 3144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:20.291396 3150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:20.291414 3149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:20.291401 3152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:20.291616 3144 server_base.cc:1034] running on GCE node
I20260924 15:18:20.291862 3144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:20.292120 3144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:20.293269 3144 hybrid_clock.cc:648] HybridClock initialized: now 1790263100293248 us; error 46 us; skew 500 ppm
Sep 24 15:18:20 dist-test-slave-zwbb krb5kdc[2901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263100, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:20.295866 3144 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:18:20.296316 3144 webserver.cc:460] Webserver started at http://127.31.12.66:41149/ using document root <none> and password file <none>
I20260924 15:18:20.296564 3144 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:20.296633 3144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:20.296797 3144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:20.297926 3144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "3ce0fc05dde647ffb038e5d74740539d"
format_stamp: "Formatted at 2026-09-24 15:18:20 on dist-test-slave-zwbb"
server_key: "a9ef3dccaa75fad92a90721a261c24fb"
server_key_iv: "789ed82714563784598be25a0fab37f9"
server_key_version: "encryptionkey@0"
I20260924 15:18:20.298292 3144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "3ce0fc05dde647ffb038e5d74740539d"
format_stamp: "Formatted at 2026-09-24 15:18:20 on dist-test-slave-zwbb"
server_key: "a9ef3dccaa75fad92a90721a261c24fb"
server_key_iv: "789ed82714563784598be25a0fab37f9"
server_key_version: "encryptionkey@0"
I20260924 15:18:20.299611 3144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:20.300426 3159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:20.300601 3144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:20.300700 3144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "3ce0fc05dde647ffb038e5d74740539d"
format_stamp: "Formatted at 2026-09-24 15:18:20 on dist-test-slave-zwbb"
server_key: "a9ef3dccaa75fad92a90721a261c24fb"
server_key_iv: "789ed82714563784598be25a0fab37f9"
server_key_version: "encryptionkey@0"
I20260924 15:18:20.300781 3144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:20.324843 3144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:20.327234 3144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:20.327404 3144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:20.327667 3144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:20.327991 3144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:20.328048 3144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:20.328089 3144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:20.328133 3144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:20.333889 3144 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:45031
I20260924 15:18:20.333981 3273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:45031 every 8 connection(s)
I20260924 15:18:20.334267 3144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:18:20.336988 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 3144
I20260924 15:18:20.337118 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:18:20.337311 31793 external_mini_cluster.cc:1428] Setting key 83c517e6805fd0f300ba58300c360ed1
Sep 24 15:18:20 dist-test-slave-zwbb krb5kdc[2901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263100, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:20.344261 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:20.334952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:39279 (local address 127.31.12.126:36707)
0924 15:18:20.334995 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:20.334997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:20.335225 (+ 228us) server_negotiation.cc:407] Connection header received
0924 15:18:20.336132 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:20.336133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:20.336142 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:20.336201 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:20.336780 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:20.337366 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:20.338580 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:20.338795 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:20.338874 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:20.341368 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:20.341369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:20.341369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:20.341377 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:20.342866 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:20.343284 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:20.343285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:20.343286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:20.343345 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:20.343611 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:20.343612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:20.343612 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:20.343733 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:20.343809 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:20.344085 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 15:18:20.344173 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:18:20.344460 3276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:20.334916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:36707 (local address 127.31.12.66:39279)
0924 15:18:20.335089 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:20.335131 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:20.336032 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:20.336272 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:20.336275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:20.336519 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:20.336709 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:20.336713 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:20.337536 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:20.337537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:20.338487 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:20.338490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:20.338880 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:20.338882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:20.339033 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:20.339574 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:20.339586 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:20.341262 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:20.342976 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:20.342982 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:20.342985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:20.343208 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:20.343434 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:20.343435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:20.343435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:20.343544 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:20.343800 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:20.343806 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:20.343946 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 15:18:20.344050 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 15:18:20.344208 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260924 15:18:20.345252 3274 heartbeater.cc:347] Connected to a master server at 127.31.12.126:36707
I20260924 15:18:20.345359 3274 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:20.345564 3274 heartbeater.cc:511] Master 127.31.12.126:36707 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.12.67@KRBTEST.COM; defaulting to no policy
I20260924 15:18:20.345986 2946 ts_manager.cc:194] Registered new tserver with Master: 3ce0fc05dde647ffb038e5d74740539d (127.31.12.66:45031)
I20260924 15:18:20.346364 2946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:39279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
I20260924 15:18:20.362504 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:36707
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:33117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:20.442615 3279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:20.442812 3279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:20.442834 3279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:20.443889 3279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:20.443923 3279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:20.443949 3279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:18:20.445392 3279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:36707
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:20.445601 3279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:20.445760 3279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:20.450789 3285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:20.450785 3284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:20.450785 3287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:20.450815 3279 server_base.cc:1034] running on GCE node
I20260924 15:18:20.451251 3279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:20.451435 3279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:20.452597 3279 hybrid_clock.cc:648] HybridClock initialized: now 1790263100452591 us; error 31 us; skew 500 ppm
Sep 24 15:18:20 dist-test-slave-zwbb krb5kdc[2901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263100, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:20.454978 3279 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:18:20.455406 3279 webserver.cc:460] Webserver started at http://127.31.12.67:38585/ using document root <none> and password file <none>
I20260924 15:18:20.455592 3279 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:20.455632 3279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:20.455715 3279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:20.456667 3279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "3762a68348964cde8292deb9dd1d8f82"
format_stamp: "Formatted at 2026-09-24 15:18:20 on dist-test-slave-zwbb"
server_key: "f1929b8eec5417ee59ffb7cbced964b8"
server_key_iv: "84bf8fc3e84618383dafcfd149a653bf"
server_key_version: "encryptionkey@0"
I20260924 15:18:20.456980 3279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "3762a68348964cde8292deb9dd1d8f82"
format_stamp: "Formatted at 2026-09-24 15:18:20 on dist-test-slave-zwbb"
server_key: "f1929b8eec5417ee59ffb7cbced964b8"
server_key_iv: "84bf8fc3e84618383dafcfd149a653bf"
server_key_version: "encryptionkey@0"
I20260924 15:18:20.458256 3279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:18:20.458963 3294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:20.459154 3279 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:20.459232 3279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "3762a68348964cde8292deb9dd1d8f82"
format_stamp: "Formatted at 2026-09-24 15:18:20 on dist-test-slave-zwbb"
server_key: "f1929b8eec5417ee59ffb7cbced964b8"
server_key_iv: "84bf8fc3e84618383dafcfd149a653bf"
server_key_version: "encryptionkey@0"
I20260924 15:18:20.459282 3279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:20.476042 3279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:20.478413 3279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:20.478524 3279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:20.478700 3279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:20.479014 3279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:20.479050 3279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:20.479072 3279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:20.479122 3279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:20.485001 3279 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:38139
I20260924 15:18:20.485083 3408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:38139 every 8 connection(s)
I20260924 15:18:20.485402 3279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:18:20.487147 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 3279
I20260924 15:18:20.487270 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:18:20.487481 31793 external_mini_cluster.cc:1428] Setting key dbb8b1a4c67e3dc473d59de1e4f34e92
Sep 24 15:18:20 dist-test-slave-zwbb krb5kdc[2901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263100, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:20.495010 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:20.486180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:44015 (local address 127.31.12.126:36707)
0924 15:18:20.486252 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:20.486254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:20.486434 (+ 180us) server_negotiation.cc:407] Connection header received
0924 15:18:20.487653 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:20.487654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:20.487668 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:20.487735 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:20.488620 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:20.489110 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:20.490105 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:20.490289 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:20.490346 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:20.492313 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:20.492314 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:20.492315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:20.492321 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:20.493754 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:20.494120 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:20.494121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:20.494121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:20.494180 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:20.494454 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:20.494454 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:20.494455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:20.494598 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:20.494650 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:20.494886 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 15:18:20.494949 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 15:18:20.495179 3411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:20.486157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:36707 (local address 127.31.12.67:44015)
0924 15:18:20.486309 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:20.486351 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:20.487498 (+ 1147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:20.487846 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:20.487850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:20.488255 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:20.488533 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:20.488545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:20.489186 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:20.489187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:20.490028 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:20.490030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:20.490362 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:20.490362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:20.490455 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:20.490904 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:20.490912 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:20.492234 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:20.493817 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:20.493822 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:20.493823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:20.494024 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:20.494260 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:20.494260 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:20.494261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:20.494353 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:20.494658 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:20.494663 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:20.494773 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 15:18:20.494855 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:18:20.494967 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260924 15:18:20.495666 3409 heartbeater.cc:347] Connected to a master server at 127.31.12.126:36707
I20260924 15:18:20.495754 3409 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:20.495890 3409 heartbeater.cc:511] Master 127.31.12.126:36707 requested a full tablet report, sending...
I20260924 15:18:20.496284 2946 ts_manager.cc:194] Registered new tserver with Master: 3762a68348964cde8292deb9dd1d8f82 (127.31.12.67:38139)
I20260924 15:18:20.496649 2946 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:44015
I20260924 15:18:20.499120 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 15:18:20 dist-test-slave-zwbb krb5kdc[2901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 15:18:20 dist-test-slave-zwbb krb5kdc[2901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.65@KRBTEST.COM
I20260924 15:18:20.521390 3416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:20.513959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42784 (local address 127.31.12.65:40457)
0924 15:18:20.514179 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:20.514184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:20.514195 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:18:20.514221 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:20.514223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:20.514236 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:20.514301 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:20.514844 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:20.515540 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:20.516225 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:20.516529 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:20.516581 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:20.518794 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:20.518802 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:20.518802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:20.518808 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:20.520376 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:20.520684 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:20.520686 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:20.520687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:20.520738 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:20.520889 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:20.520890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:20.520890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:20.521026 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:20.521123 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:20.521189 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 15:18:20.521253 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":129,"threads_started":1}
Sep 24 15:18:20 dist-test-slave-zwbb krb5kdc[2901](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:20.529599 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:20.522639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59130 (local address 127.31.12.126:36707)
0924 15:18:20.522691 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:20.522693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:20.522723 (+ 30us) server_negotiation.cc:407] Connection header received
0924 15:18:20.522799 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:20.522800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:20.522810 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:20.522883 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:20.523370 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:20.524235 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:20.525013 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:20.525201 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:20.525258 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:20.527264 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:20.527265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:20.527265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:20.527271 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:20.528618 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:20.528893 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:20.528894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:20.528894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:20.528956 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:20.529115 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:20.529115 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:20.529116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:20.529240 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:20.529309 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:20.529392 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 15:18:20.529501 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 15:18:20.530512 2946 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59130
I20260924 15:18:20.530877 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 3009
I20260924 15:18:20.536365 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 3144
I20260924 15:18:20.541379 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 3279
I20260924 15:18:20.546034 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 2914
2026-09-24T15:18:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1848 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:18:20 dist-test-slave-zwbb krb5kdc[3421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:18:20 dist-test-slave-zwbb krb5kdc[3421](info): set up 2 sockets
Sep 24 15:18:20 dist-test-slave-zwbb krb5kdc[3421](info): commencing operation
krb5kdc: starting...
W20260924 15:18:21.595827 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:18:21 dist-test-slave-zwbb krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:18:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:18:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:18:21.713807 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:34527
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:33803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:34527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:21.798143 3435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:21.798372 3435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:21.798396 3435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:21.799659 3435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:18:21.799700 3435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:21.799714 3435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:21.799727 3435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:18:21.799758 3435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:18:21.801321 3435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:34527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:34527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:21.801641 3435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:21.801831 3435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:21.806870 3441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:21.806871 3440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:21.807065 3443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:21.807065 3435 server_base.cc:1034] running on GCE node
I20260924 15:18:21.807348 3435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:21.807615 3435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:21.808748 3435 hybrid_clock.cc:648] HybridClock initialized: now 1790263101808720 us; error 47 us; skew 500 ppm
Sep 24 15:18:21 dist-test-slave-zwbb krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263101, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:21.811136 3435 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:18:21.811479 3435 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:21.811554 3435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:21.811707 3435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:21.812757 3435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "77e59abc0a8145cf991c4818f4ec3525"
format_stamp: "Formatted at 2026-09-24 15:18:21 on dist-test-slave-zwbb"
server_key: "a1a17eebaa3092c82d1707b28fb50d03"
server_key_iv: "62310088955d875027815eb3d4bdd63e"
server_key_version: "encryptionkey@0"
I20260924 15:18:21.813123 3435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "77e59abc0a8145cf991c4818f4ec3525"
format_stamp: "Formatted at 2026-09-24 15:18:21 on dist-test-slave-zwbb"
server_key: "a1a17eebaa3092c82d1707b28fb50d03"
server_key_iv: "62310088955d875027815eb3d4bdd63e"
server_key_version: "encryptionkey@0"
I20260924 15:18:21.814494 3435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:18:21.815222 3449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:21.815393 3435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:18:21.815495 3435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "77e59abc0a8145cf991c4818f4ec3525"
format_stamp: "Formatted at 2026-09-24 15:18:21 on dist-test-slave-zwbb"
server_key: "a1a17eebaa3092c82d1707b28fb50d03"
server_key_iv: "62310088955d875027815eb3d4bdd63e"
server_key_version: "encryptionkey@0"
I20260924 15:18:21.815579 3435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:21.844467 3435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:21.846462 3435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:21.846634 3435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:21.850639 3435 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:34527
I20260924 15:18:21.850683 3501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:34527 every 8 connection(s)
I20260924 15:18:21.851006 3435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:18:21.851974 3502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:21.854435 3502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525: Bootstrap starting.
I20260924 15:18:21.855024 3502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:21.855332 3502 log.cc:824] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:21.856007 3502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525: No bootstrap required, opened a new log
I20260924 15:18:21.857630 3502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e59abc0a8145cf991c4818f4ec3525" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 34527 } }
I20260924 15:18:21.857765 3502 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:21.857813 3502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e59abc0a8145cf991c4818f4ec3525, State: Initialized, Role: FOLLOWER
I20260924 15:18:21.857978 3502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e59abc0a8145cf991c4818f4ec3525" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 34527 } }
I20260924 15:18:21.858040 3502 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:18:21.858081 3502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:18:21.858129 3502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:21.858680 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 3435
I20260924 15:18:21.858771 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:18:21.858947 31793 external_mini_cluster.cc:1428] Setting key 8b8b54c1801ab8e2073d2d98a59f2729
I20260924 15:18:21.859010 3502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e59abc0a8145cf991c4818f4ec3525" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 34527 } }
I20260924 15:18:21.859134 3502 leader_election.cc:302] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77e59abc0a8145cf991c4818f4ec3525; no voters:
I20260924 15:18:21.859346 3502 leader_election.cc:288] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:18:21.859385 3505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:21.859549 3505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525 [term 1 LEADER]: Becoming Leader. State: Replica: 77e59abc0a8145cf991c4818f4ec3525, State: Running, Role: LEADER
I20260924 15:18:21.859632 3502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:18:21.859643 3505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e59abc0a8145cf991c4818f4ec3525" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 34527 } }
I20260924 15:18:21.860258 3509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77e59abc0a8145cf991c4818f4ec3525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e59abc0a8145cf991c4818f4ec3525" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 34527 } } }
I20260924 15:18:21.860360 3509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525 [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:21.860531 3507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77e59abc0a8145cf991c4818f4ec3525. Latest consensus state: current_term: 1 leader_uuid: "77e59abc0a8145cf991c4818f4ec3525" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e59abc0a8145cf991c4818f4ec3525" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 34527 } } }
I20260924 15:18:21.860617 3507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525 [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:21.860569 3511 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:18:21.861188 3511 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:18:21.863375 3511 catalog_manager.cc:1347] Generated new cluster ID: f5751ef46aaf4b9da4640470cb491b70
I20260924 15:18:21.863431 3511 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:18:21 dist-test-slave-zwbb krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:21.870563 3508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:21.859409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.31.12.126:34527)
0924 15:18:21.859690 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:21.859699 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:21.859728 (+ 29us) server_negotiation.cc:407] Connection header received
0924 15:18:21.859893 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:21.859901 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:21.859963 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:21.860145 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:21.861004 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:21.862457 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:21.863317 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:21.863904 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:21.864010 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:21.866809 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:21.866813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:21.866813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:21.866824 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:21.868852 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:21.869242 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:21.869245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:21.869246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:21.869390 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:21.869625 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:21.869626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:21.869626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:21.869949 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:21.870127 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:21.870205 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 15:18:21.870349 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":94,"threads_started":1}
I20260924 15:18:21.873314 3511 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:18:21.874051 3511 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:18:21.888542 3511 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525: Generated new TSK 0
I20260924 15:18:21.888741 3511 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
I20260924 15:18:21.918645 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:34527
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:33803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:21.999774 3528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:21.999975 3528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:22.000000 3528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:22.001178 3528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:22.001223 3528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:22.001251 3528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:18:22.002787 3528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:34527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:22.003077 3528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:22.003324 3528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:22.008390 3533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:22.008436 3536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:22.008399 3534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:22.008431 3528 server_base.cc:1034] running on GCE node
I20260924 15:18:22.008764 3528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:22.008997 3528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:22.010160 3528 hybrid_clock.cc:648] HybridClock initialized: now 1790263102010143 us; error 32 us; skew 500 ppm
Sep 24 15:18:22 dist-test-slave-zwbb krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263102, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:22.012615 3528 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:18:22.012971 3528 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:22.013181 3528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:22.013520 3528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:22.014606 3528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "6f23ffe289a543518b472f6fd28d7ef3"
format_stamp: "Formatted at 2026-09-24 15:18:22 on dist-test-slave-zwbb"
server_key: "459930b72d4f84715cbb77f43be674af"
server_key_iv: "a7522cb2f1fc817d9a9bf57313a0fb57"
server_key_version: "encryptionkey@0"
I20260924 15:18:22.014972 3528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "6f23ffe289a543518b472f6fd28d7ef3"
format_stamp: "Formatted at 2026-09-24 15:18:22 on dist-test-slave-zwbb"
server_key: "459930b72d4f84715cbb77f43be674af"
server_key_iv: "a7522cb2f1fc817d9a9bf57313a0fb57"
server_key_version: "encryptionkey@0"
I20260924 15:18:22.016301 3528 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:18:22.017057 3542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:22.017223 3528 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:22.017328 3528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "6f23ffe289a543518b472f6fd28d7ef3"
format_stamp: "Formatted at 2026-09-24 15:18:22 on dist-test-slave-zwbb"
server_key: "459930b72d4f84715cbb77f43be674af"
server_key_iv: "a7522cb2f1fc817d9a9bf57313a0fb57"
server_key_version: "encryptionkey@0"
I20260924 15:18:22.017407 3528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:22.032871 3528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:22.034657 3528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:22.034824 3528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:22.035064 3528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:22.035374 3528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:22.035427 3528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:22.035460 3528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:22.035490 3528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:22.041113 3528 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:44927
I20260924 15:18:22.041147 3656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:44927 every 8 connection(s)
I20260924 15:18:22.041421 3528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:18:22.043515 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 3528
I20260924 15:18:22.043655 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:18:22.043844 31793 external_mini_cluster.cc:1428] Setting key 6fb31a9d0765ae5b76915dde11cc5e85
Sep 24 15:18:22 dist-test-slave-zwbb krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263102, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.12.66@KRBTEST.COM; defaulting to no policy
I20260924 15:18:22.052816 3508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.042202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:42909 (local address 127.31.12.126:34527)
0924 15:18:22.042266 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:22.042268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:22.042469 (+ 201us) server_negotiation.cc:407] Connection header received
0924 15:18:22.042949 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:22.042950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:22.042963 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:22.043022 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:22.045306 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.045892 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.046916 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.047239 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.047313 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:22.049614 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:22.049616 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:22.049617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:22.049627 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:22.051363 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:22.051780 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:22.051782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:22.051782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:22.051852 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:22.052111 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:22.052112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:22.052113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:22.052268 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:22.052353 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:22.052600 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 15:18:22.052699 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:18:22.052940 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.042134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:34527 (local address 127.31.12.65:42909)
0924 15:18:22.042337 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:22.042379 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:22.042849 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:22.043114 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:22.043118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:22.043374 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:22.045191 (+ 1817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:22.045198 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.046009 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.046011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:22.046816 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:22.046818 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.047327 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.047328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:22.047440 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:22.047991 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:22.048000 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:22.049497 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:22.051465 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:22.051473 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:22.051475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:22.051700 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:22.051926 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:22.051927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:22.051927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:22.052037 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:22.052344 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:22.052349 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:22.052463 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 15:18:22.052551 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 15:18:22.052710 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":126,"threads_started":1}
I20260924 15:18:22.053551 3657 heartbeater.cc:347] Connected to a master server at 127.31.12.126:34527
I20260924 15:18:22.053661 3657 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:22.053831 3657 heartbeater.cc:511] Master 127.31.12.126:34527 requested a full tablet report, sending...
I20260924 15:18:22.054415 3466 ts_manager.cc:194] Registered new tserver with Master: 6f23ffe289a543518b472f6fd28d7ef3 (127.31.12.65:44927)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.66@KRBTEST.COM" created.
I20260924 15:18:22.055219 3466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:42909
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
I20260924 15:18:22.069796 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:34527
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:33803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:22.149622 3663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:22.149813 3663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:22.149837 3663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:22.150903 3663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:22.150943 3663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:22.150969 3663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:18:22.152369 3663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:34527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:22.152561 3663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:22.152733 3663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:22.157500 3669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:22.157542 3668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:22.157583 3671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:22.157634 3663 server_base.cc:1034] running on GCE node
I20260924 15:18:22.157861 3663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:22.158099 3663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:22.159257 3663 hybrid_clock.cc:648] HybridClock initialized: now 1790263102159245 us; error 42 us; skew 500 ppm
Sep 24 15:18:22 dist-test-slave-zwbb krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263102, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:22.161896 3663 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:18:22.162277 3663 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:22.162346 3663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:22.162488 3663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:22.163503 3663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "03bb374658c8465481e991e78007a38b"
format_stamp: "Formatted at 2026-09-24 15:18:22 on dist-test-slave-zwbb"
server_key: "f6d578e4f317359676a19b6c795905e5"
server_key_iv: "f439455d3f94d8d8c0d85c878f770269"
server_key_version: "encryptionkey@0"
I20260924 15:18:22.163868 3663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "03bb374658c8465481e991e78007a38b"
format_stamp: "Formatted at 2026-09-24 15:18:22 on dist-test-slave-zwbb"
server_key: "f6d578e4f317359676a19b6c795905e5"
server_key_iv: "f439455d3f94d8d8c0d85c878f770269"
server_key_version: "encryptionkey@0"
I20260924 15:18:22.165169 3663 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:18:22.165948 3677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:22.166131 3663 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:22.166229 3663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "03bb374658c8465481e991e78007a38b"
format_stamp: "Formatted at 2026-09-24 15:18:22 on dist-test-slave-zwbb"
server_key: "f6d578e4f317359676a19b6c795905e5"
server_key_iv: "f439455d3f94d8d8c0d85c878f770269"
server_key_version: "encryptionkey@0"
I20260924 15:18:22.166313 3663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:22.179342 3663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:22.181078 3663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:22.181234 3663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:22.181493 3663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:22.181804 3663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:22.181859 3663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:22.181892 3663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:22.181919 3663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:22.187691 3663 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:33413
I20260924 15:18:22.187731 3791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:33413 every 8 connection(s)
I20260924 15:18:22.187979 3663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:18:22 dist-test-slave-zwbb krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263102, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:22.194306 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 3663
I20260924 15:18:22.194416 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:18:22.194662 31793 external_mini_cluster.cc:1428] Setting key dcff52ced93d1fbc5c8bb14653732fcf
I20260924 15:18:22.197691 3508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.188682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:38373 (local address 127.31.12.126:34527)
0924 15:18:22.188750 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:22.188752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:22.188964 (+ 212us) server_negotiation.cc:407] Connection header received
0924 15:18:22.189395 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:22.189396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:22.189409 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:22.189526 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:22.190443 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.191040 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.192014 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.192214 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.192272 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:22.194616 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:22.194617 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:22.194618 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:22.194624 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:22.196086 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:22.196599 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:22.196600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:22.196601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:22.196670 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:22.196950 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:22.196951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:22.196951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:22.197100 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:22.197186 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:22.197421 (+ 235us) server_negotiation.cc:299] Negotiation successful
0924 15:18:22.197564 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 15:18:22.197731 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.188642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:34527 (local address 127.31.12.66:38373)
0924 15:18:22.188803 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:22.188845 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:22.189315 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:22.189619 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:22.189622 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:22.189877 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:22.190361 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:22.190366 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.191134 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.191135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:22.191931 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:22.191933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.192289 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.192290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:22.192388 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:22.192948 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:22.192955 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:22.194519 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:22.196171 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:22.196179 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:22.196181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:22.196516 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:22.196752 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:22.196753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:22.196754 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:22.196884 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:22.197180 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:22.197185 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:22.197292 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 15:18:22.197367 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 15:18:22.197577 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":88,"threads_started":1}
I20260924 15:18:22.198320 3792 heartbeater.cc:347] Connected to a master server at 127.31.12.126:34527
I20260924 15:18:22.198407 3792 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:22.198568 3792 heartbeater.cc:511] Master 127.31.12.126:34527 requested a full tablet report, sending...
I20260924 15:18:22.198982 3466 ts_manager.cc:194] Registered new tserver with Master: 03bb374658c8465481e991e78007a38b (127.31.12.66:33413)
I20260924 15:18:22.199514 3466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:38373
WARNING: no policy specified for kudu/127.31.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
I20260924 15:18:22.219872 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:34527
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:33803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:22.299085 3797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:22.299283 3797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:22.299306 3797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:22.300449 3797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:22.300491 3797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:22.300547 3797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:18:22.302091 3797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:34527
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:22.302321 3797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:22.302541 3797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:18:22.307367 3797 server_base.cc:1034] running on GCE node
W20260924 15:18:22.307387 3803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:22.307380 3802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:22.307542 3805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:22.307771 3797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:22.307965 3797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:22.309150 3797 hybrid_clock.cc:648] HybridClock initialized: now 1790263102309138 us; error 29 us; skew 500 ppm
Sep 24 15:18:22 dist-test-slave-zwbb krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263102, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:22.311719 3797 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:18:22.312099 3797 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:22.312150 3797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:22.312307 3797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:22.313331 3797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "27358c8ff08b41fbbd7980ccd096b0bf"
format_stamp: "Formatted at 2026-09-24 15:18:22 on dist-test-slave-zwbb"
server_key: "90fba5aa2c82c63372d2f3172d3f9619"
server_key_iv: "e6d6bb31baaf00b19c5a6a82a98d10fc"
server_key_version: "encryptionkey@0"
I20260924 15:18:22.313755 3797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "27358c8ff08b41fbbd7980ccd096b0bf"
format_stamp: "Formatted at 2026-09-24 15:18:22 on dist-test-slave-zwbb"
server_key: "90fba5aa2c82c63372d2f3172d3f9619"
server_key_iv: "e6d6bb31baaf00b19c5a6a82a98d10fc"
server_key_version: "encryptionkey@0"
I20260924 15:18:22.315019 3797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:22.315826 3811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:22.316018 3797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:22.316100 3797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "27358c8ff08b41fbbd7980ccd096b0bf"
format_stamp: "Formatted at 2026-09-24 15:18:22 on dist-test-slave-zwbb"
server_key: "90fba5aa2c82c63372d2f3172d3f9619"
server_key_iv: "e6d6bb31baaf00b19c5a6a82a98d10fc"
server_key_version: "encryptionkey@0"
I20260924 15:18:22.316171 3797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:22.343324 3797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:22.345153 3797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:22.345307 3797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:22.345589 3797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:22.345921 3797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:22.345976 3797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:22.346019 3797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:22.346046 3797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:22.352067 3797 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:34707
I20260924 15:18:22.352105 3925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:34707 every 8 connection(s)
I20260924 15:18:22.352353 3797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:18:22.354867 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 3797
I20260924 15:18:22.354990 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:18:22.355192 31793 external_mini_cluster.cc:1428] Setting key bad18f8006a8ec1958f8d93d0715bc33
Sep 24 15:18:22 dist-test-slave-zwbb krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263102, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:22.361749 3508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.353049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:36425 (local address 127.31.12.126:34527)
0924 15:18:22.353097 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:22.353099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:22.353312 (+ 213us) server_negotiation.cc:407] Connection header received
0924 15:18:22.353795 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:22.353796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:22.353807 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:22.353876 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:22.354840 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.355649 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.356718 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.356894 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.356955 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:22.358936 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:22.358937 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:22.358938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:22.358944 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:22.360492 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:22.360853 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:22.360854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:22.360854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:22.360903 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:22.361084 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:22.361084 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:22.361084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:22.361168 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:22.361226 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:22.361563 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 15:18:22.361667 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:18:22.361907 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.353015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:34527 (local address 127.31.12.67:36425)
0924 15:18:22.353170 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:22.353220 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:22.353715 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:22.353930 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:22.353933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:22.354185 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:22.354744 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:22.354752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.355734 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.355735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:22.356649 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:22.356651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.356958 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.356959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:22.357046 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:22.357499 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:22.357508 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:22.358842 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:22.360558 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:22.360564 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:22.360566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:22.360791 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:22.360960 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:22.360960 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:22.360961 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:22.361047 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:22.361212 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:22.361216 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:22.361367 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 15:18:22.361536 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 15:18:22.361704 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260924 15:18:22.362362 3926 heartbeater.cc:347] Connected to a master server at 127.31.12.126:34527
I20260924 15:18:22.362457 3926 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:22.362599 3926 heartbeater.cc:511] Master 127.31.12.126:34527 requested a full tablet report, sending...
I20260924 15:18:22.362967 3466 ts_manager.cc:194] Registered new tserver with Master: 27358c8ff08b41fbbd7980ccd096b0bf (127.31.12.67:34707)
I20260924 15:18:22.363322 3466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:36425
I20260924 15:18:22.366848 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:18:22.373966 3508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.368117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55318 (local address 127.31.12.126:34527)
0924 15:18:22.368177 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:22.368179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:22.368302 (+ 123us) server_negotiation.cc:407] Connection header received
0924 15:18:22.368349 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:22.368355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:22.368363 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:22.368400 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:22.368969 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.369420 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.370160 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.370320 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.370387 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:22.371525 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:22.371527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:22.371527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:22.371532 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:22.372871 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:22.373176 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:22.373177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:22.373178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:22.373273 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:22.373430 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:22.373431 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:22.373431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:22.373621 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:22.373703 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:22.373792 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 15:18:22.373853 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 15:18:22.375384 3466 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:18:22.376855 3466 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:18:22.383255 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.379053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:33413 (local address 127.0.0.1:33648)
0924 15:18:22.379378 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:22.379412 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:22.379475 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:22.379771 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:22.379772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:22.379774 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:22.379827 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:22.379828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.381161 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.381162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:22.382274 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:22.382275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.382958 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.382959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:22.383050 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:22.383064 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 15:18:22.383132 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 15:18:22.383183 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":1,"thread_start_us":51,"threads_started":1}
I20260924 15:18:22.383499 3937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.379221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33648 (local address 127.31.12.66:33413)
0924 15:18:22.379608 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:22.379612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:22.379620 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:18:22.379650 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:22.379652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:22.379662 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:22.379931 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:22.380013 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.381070 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.382351 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.382830 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.382954 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:22.383018 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:22.383362 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 15:18:22.383421 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":72,"threads_started":1}
I20260924 15:18:22.384136 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.379239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:34707 (local address 127.0.0.1:45190)
0924 15:18:22.379840 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:22.379850 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:22.379898 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:22.380087 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:22.380088 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:22.380090 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:22.380132 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:22.380134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.380945 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.380946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:22.382270 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:22.382274 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.383877 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.383878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:22.383940 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:22.383945 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:18:22.384012 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 15:18:22.384060 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":47,"threads_started":1}
I20260924 15:18:22.384215 3725 tablet_service.cc:1467] Processing CreateTablet for tablet c6ee64d0bd0947adbd80bbfd0ce07b2c (DEFAULT_TABLE table=test-table [id=4278f26e490642149fa4c8735fdeebec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:22.384347 3938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.379329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45190 (local address 127.31.12.67:34707)
0924 15:18:22.379618 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:22.379621 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:22.379954 (+ 333us) server_negotiation.cc:407] Connection header received
0924 15:18:22.379995 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:22.379998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:22.380013 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:22.380090 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:22.380187 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.380845 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.383055 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.383769 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.383878 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:22.383934 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:22.384232 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 15:18:22.384271 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":100,"threads_started":1}
I20260924 15:18:22.384533 3936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.379046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:44927 (local address 127.0.0.1:56758)
0924 15:18:22.379362 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:22.379406 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:22.379476 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:22.379752 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:22.379755 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:22.379757 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:22.379822 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:22.379824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.382373 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.382374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:22.383028 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:22.383029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.384287 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.384288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:22.384349 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:22.384352 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:18:22.384414 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 15:18:22.384451 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":53,"threads_started":1}
I20260924 15:18:22.384570 3725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ee64d0bd0947adbd80bbfd0ce07b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:22.384567 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.379373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.31.12.65:44927)
0924 15:18:22.379559 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:22.379565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:22.379576 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:18:22.379617 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:22.379619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:22.379636 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:22.379735 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:22.379914 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.381046 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.383129 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.384120 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.384194 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:22.384248 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:22.384454 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 15:18:22.384500 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":77,"threads_started":1}
I20260924 15:18:22.385146 3591 tablet_service.cc:1467] Processing CreateTablet for tablet c6ee64d0bd0947adbd80bbfd0ce07b2c (DEFAULT_TABLE table=test-table [id=4278f26e490642149fa4c8735fdeebec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:22.385375 3591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ee64d0bd0947adbd80bbfd0ce07b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:22.386098 3850 tablet_service.cc:1467] Processing CreateTablet for tablet c6ee64d0bd0947adbd80bbfd0ce07b2c (DEFAULT_TABLE table=test-table [id=4278f26e490642149fa4c8735fdeebec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:22.386420 3850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6ee64d0bd0947adbd80bbfd0ce07b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:22.387817 3940 tablet_bootstrap.cc:492] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b: Bootstrap starting.
I20260924 15:18:22.388245 3941 tablet_bootstrap.cc:492] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 6f23ffe289a543518b472f6fd28d7ef3: Bootstrap starting.
I20260924 15:18:22.388598 3940 tablet_bootstrap.cc:654] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:22.388855 3941 tablet_bootstrap.cc:654] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 6f23ffe289a543518b472f6fd28d7ef3: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:22.389194 3940 log.cc:824] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:22.389484 3941 log.cc:824] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 6f23ffe289a543518b472f6fd28d7ef3: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:22.389787 3943 tablet_bootstrap.cc:492] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf: Bootstrap starting.
I20260924 15:18:22.389999 3940 tablet_bootstrap.cc:492] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b: No bootstrap required, opened a new log
I20260924 15:18:22.390082 3940 ts_tablet_manager.cc:1397] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:18:22.390297 3941 tablet_bootstrap.cc:492] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 6f23ffe289a543518b472f6fd28d7ef3: No bootstrap required, opened a new log
I20260924 15:18:22.390379 3941 ts_tablet_manager.cc:1397] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 6f23ffe289a543518b472f6fd28d7ef3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:18:22.390471 3943 tablet_bootstrap.cc:654] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:22.391024 3943 log.cc:824] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:22.391921 3943 tablet_bootstrap.cc:492] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf: No bootstrap required, opened a new log
I20260924 15:18:22.391995 3943 ts_tablet_manager.cc:1397] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:18:22.392118 3941 raft_consensus.cc:348] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 6f23ffe289a543518b472f6fd28d7ef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bb374658c8465481e991e78007a38b" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33413 } } peers { permanent_uuid: "27358c8ff08b41fbbd7980ccd096b0bf" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 34707 } } peers { permanent_uuid: "6f23ffe289a543518b472f6fd28d7ef3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 44927 } }
I20260924 15:18:22.392282 3941 raft_consensus.cc:374] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 6f23ffe289a543518b472f6fd28d7ef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:22.392326 3941 raft_consensus.cc:729] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 6f23ffe289a543518b472f6fd28d7ef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f23ffe289a543518b472f6fd28d7ef3, State: Initialized, Role: FOLLOWER
I20260924 15:18:22.392439 3941 consensus_queue.cc:260] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 6f23ffe289a543518b472f6fd28d7ef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bb374658c8465481e991e78007a38b" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33413 } } peers { permanent_uuid: "27358c8ff08b41fbbd7980ccd096b0bf" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 34707 } } peers { permanent_uuid: "6f23ffe289a543518b472f6fd28d7ef3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 44927 } }
I20260924 15:18:22.392732 3941 ts_tablet_manager.cc:1428] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 6f23ffe289a543518b472f6fd28d7ef3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 15:18:22.392678 3940 raft_consensus.cc:348] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bb374658c8465481e991e78007a38b" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33413 } } peers { permanent_uuid: "27358c8ff08b41fbbd7980ccd096b0bf" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 34707 } } peers { permanent_uuid: "6f23ffe289a543518b472f6fd28d7ef3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 44927 } }
I20260924 15:18:22.392846 3940 raft_consensus.cc:374] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:22.392866 3657 heartbeater.cc:503] Master 127.31.12.126:34527 was elected leader, sending a full tablet report...
I20260924 15:18:22.392897 3940 raft_consensus.cc:729] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03bb374658c8465481e991e78007a38b, State: Initialized, Role: FOLLOWER
I20260924 15:18:22.393004 3940 consensus_queue.cc:260] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bb374658c8465481e991e78007a38b" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33413 } } peers { permanent_uuid: "27358c8ff08b41fbbd7980ccd096b0bf" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 34707 } } peers { permanent_uuid: "6f23ffe289a543518b472f6fd28d7ef3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 44927 } }
I20260924 15:18:22.393249 3940 ts_tablet_manager.cc:1428] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:18:22.393287 3792 heartbeater.cc:503] Master 127.31.12.126:34527 was elected leader, sending a full tablet report...
I20260924 15:18:22.394061 3943 raft_consensus.cc:348] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bb374658c8465481e991e78007a38b" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33413 } } peers { permanent_uuid: "27358c8ff08b41fbbd7980ccd096b0bf" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 34707 } } peers { permanent_uuid: "6f23ffe289a543518b472f6fd28d7ef3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 44927 } }
I20260924 15:18:22.394215 3943 raft_consensus.cc:374] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:22.394255 3943 raft_consensus.cc:729] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27358c8ff08b41fbbd7980ccd096b0bf, State: Initialized, Role: FOLLOWER
I20260924 15:18:22.394361 3943 consensus_queue.cc:260] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bb374658c8465481e991e78007a38b" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33413 } } peers { permanent_uuid: "27358c8ff08b41fbbd7980ccd096b0bf" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 34707 } } peers { permanent_uuid: "6f23ffe289a543518b472f6fd28d7ef3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 44927 } }
I20260924 15:18:22.394647 3943 ts_tablet_manager.cc:1428] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:18:22.394647 3926 heartbeater.cc:503] Master 127.31.12.126:34527 was elected leader, sending a full tablet report...
W20260924 15:18:22.438484 3793 tablet.cc:2369] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:18:22.542141 3658 tablet.cc:2369] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 6f23ffe289a543518b472f6fd28d7ef3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:18:22.602895 3927 tablet.cc:2369] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:18:22.606213 3948 raft_consensus.cc:482] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:18:22.606350 3948 raft_consensus.cc:504] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bb374658c8465481e991e78007a38b" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33413 } } peers { permanent_uuid: "27358c8ff08b41fbbd7980ccd096b0bf" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 34707 } } peers { permanent_uuid: "6f23ffe289a543518b472f6fd28d7ef3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 44927 } }
I20260924 15:18:22.606724 3948 leader_election.cc:288] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 03bb374658c8465481e991e78007a38b (127.31.12.66:33413), 6f23ffe289a543518b472f6fd28d7ef3 (127.31.12.65:44927)
I20260924 15:18:22.609486 3928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.606907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:44927 (local address 127.31.12.67:58463)
0924 15:18:22.606959 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:22.606970 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:22.607033 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:22.607196 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:22.607197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:22.607199 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:22.607267 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:22.607269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.607910 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.607911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:22.608657 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:22.608659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.609198 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.609198 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:22.609289 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:22.609292 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:18:22.609354 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 15:18:22.609394 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 15:18:22.609485 3939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.607003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:58463 (local address 127.31.12.65:44927)
0924 15:18:22.607060 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:22.607062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:22.607072 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:18:22.607103 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:22.607104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:22.607111 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:22.607187 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:22.607347 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.607824 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.608754 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.609114 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.609198 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:22.609206 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:22.609372 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 15:18:22.609406 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:18:22.609817 3949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.606916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:33413 (local address 127.31.12.67:53653)
0924 15:18:22.607255 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:22.607264 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:22.607341 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:22.607486 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:22.607487 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:22.607488 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:22.607558 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:22.607560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.608174 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.608175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:22.608916 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:22.608918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.609553 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.609554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:22.609626 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:22.609630 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:18:22.609703 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 15:18:22.609740 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":170,"thread_start_us":121,"threads_started":1}
I20260924 15:18:22.609810 3937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.607005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:53653 (local address 127.31.12.66:33413)
0924 15:18:22.607071 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:22.607073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:22.607329 (+ 256us) server_negotiation.cc:407] Connection header received
0924 15:18:22.607405 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:22.607406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:22.607414 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:22.607483 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:22.607636 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.608091 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.609030 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.609410 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.609538 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:22.609547 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:22.609712 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 15:18:22.609751 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:18:22.609817 3611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ee64d0bd0947adbd80bbfd0ce07b2c" candidate_uuid: "27358c8ff08b41fbbd7980ccd096b0bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f23ffe289a543518b472f6fd28d7ef3" is_pre_election: true
I20260924 15:18:22.609961 3611 raft_consensus.cc:2454] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 6f23ffe289a543518b472f6fd28d7ef3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27358c8ff08b41fbbd7980ccd096b0bf in term 0.
I20260924 15:18:22.610210 3815 leader_election.cc:302] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27358c8ff08b41fbbd7980ccd096b0bf, 6f23ffe289a543518b472f6fd28d7ef3; no voters:
I20260924 15:18:22.610344 3948 raft_consensus.cc:2781] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:18:22.610342 3745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ee64d0bd0947adbd80bbfd0ce07b2c" candidate_uuid: "27358c8ff08b41fbbd7980ccd096b0bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03bb374658c8465481e991e78007a38b" is_pre_election: true
I20260924 15:18:22.610399 3948 raft_consensus.cc:482] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:18:22.610457 3948 raft_consensus.cc:3037] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:22.610495 3745 raft_consensus.cc:2454] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 27358c8ff08b41fbbd7980ccd096b0bf in term 0.
I20260924 15:18:22.611203 3948 raft_consensus.cc:504] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bb374658c8465481e991e78007a38b" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33413 } } peers { permanent_uuid: "27358c8ff08b41fbbd7980ccd096b0bf" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 34707 } } peers { permanent_uuid: "6f23ffe289a543518b472f6fd28d7ef3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 44927 } }
I20260924 15:18:22.611348 3948 leader_election.cc:288] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [CANDIDATE]: Term 1 election: Requested vote from peers 03bb374658c8465481e991e78007a38b (127.31.12.66:33413), 6f23ffe289a543518b472f6fd28d7ef3 (127.31.12.65:44927)
I20260924 15:18:22.611475 3745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ee64d0bd0947adbd80bbfd0ce07b2c" candidate_uuid: "27358c8ff08b41fbbd7980ccd096b0bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03bb374658c8465481e991e78007a38b"
I20260924 15:18:22.611524 3611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6ee64d0bd0947adbd80bbfd0ce07b2c" candidate_uuid: "27358c8ff08b41fbbd7980ccd096b0bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f23ffe289a543518b472f6fd28d7ef3"
I20260924 15:18:22.611578 3745 raft_consensus.cc:3037] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:22.611589 3611 raft_consensus.cc:3037] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 6f23ffe289a543518b472f6fd28d7ef3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:22.612450 3611 raft_consensus.cc:2454] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 6f23ffe289a543518b472f6fd28d7ef3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27358c8ff08b41fbbd7980ccd096b0bf in term 1.
I20260924 15:18:22.612450 3745 raft_consensus.cc:2454] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 27358c8ff08b41fbbd7980ccd096b0bf in term 1.
I20260924 15:18:22.612619 3815 leader_election.cc:302] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27358c8ff08b41fbbd7980ccd096b0bf, 6f23ffe289a543518b472f6fd28d7ef3; no voters:
I20260924 15:18:22.612720 3948 raft_consensus.cc:2781] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:22.612848 3948 raft_consensus.cc:686] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [term 1 LEADER]: Becoming Leader. State: Replica: 27358c8ff08b41fbbd7980ccd096b0bf, State: Running, Role: LEADER
I20260924 15:18:22.612948 3948 consensus_queue.cc:237] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bb374658c8465481e991e78007a38b" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33413 } } peers { permanent_uuid: "27358c8ff08b41fbbd7980ccd096b0bf" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 34707 } } peers { permanent_uuid: "6f23ffe289a543518b472f6fd28d7ef3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 44927 } }
I20260924 15:18:22.613726 3466 catalog_manager.cc:5495] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf reported cstate change: term changed from 0 to 1, leader changed from <none> to 27358c8ff08b41fbbd7980ccd096b0bf (127.31.12.67). New cstate: current_term: 1 leader_uuid: "27358c8ff08b41fbbd7980ccd096b0bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "03bb374658c8465481e991e78007a38b" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "27358c8ff08b41fbbd7980ccd096b0bf" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 34707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f23ffe289a543518b472f6fd28d7ef3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 44927 } health_report { overall_health: UNKNOWN } } }
I20260924 15:18:22.666136 3938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.663632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45206 (local address 127.31.12.67:34707)
0924 15:18:22.663705 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:22.663707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:22.663716 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:18:22.663760 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:22.663761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:22.663784 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:22.663870 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:18:22.664026 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.664538 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.665317 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:22.665541 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:22.665619 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:22.665787 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:18:22.665896 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:18:22.665992 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:22.666012 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 15:18:22.666031 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 15:18:22.668033 3745 raft_consensus.cc:1261] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b [term 1 FOLLOWER]: Refusing update from remote peer 27358c8ff08b41fbbd7980ccd096b0bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:22.668076 3611 raft_consensus.cc:1261] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 6f23ffe289a543518b472f6fd28d7ef3 [term 1 FOLLOWER]: Refusing update from remote peer 27358c8ff08b41fbbd7980ccd096b0bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:22.668462 3948 consensus_queue.cc:1035] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f23ffe289a543518b472f6fd28d7ef3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 44927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:22.668620 3950 consensus_queue.cc:1035] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "03bb374658c8465481e991e78007a38b" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:22.671373 3954 mvcc.cc:205] Tried to move back new op lower bound from 7332917668525506560 to 7332917668303241216. Current Snapshot: MvccSnapshot[applied={T|T < 7332917668525506560}]
I20260924 15:18:22.671468 3955 mvcc.cc:205] Tried to move back new op lower bound from 7332917668525506560 to 7332917668303241216. Current Snapshot: MvccSnapshot[applied={T|T < 7332917668525506560}]
I20260924 15:18:22.673601 3465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:18:22.673693 3465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:18:22.674500 3465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77e59abc0a8145cf991c4818f4ec3525: Sending DeleteTablet for 3 replicas of tablet c6ee64d0bd0947adbd80bbfd0ce07b2c
I20260924 15:18:22.674836 3850 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ee64d0bd0947adbd80bbfd0ce07b2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:22 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:45190
I20260924 15:18:22.674862 3725 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ee64d0bd0947adbd80bbfd0ce07b2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:22 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:33648
I20260924 15:18:22.675022 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 3528
I20260924 15:18:22.675071 3591 tablet_service.cc:1514] Processing DeleteTablet for tablet c6ee64d0bd0947adbd80bbfd0ce07b2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:22 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:56758
I20260924 15:18:22.675554 3964 tablet_replica.cc:330] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf: stopping tablet replica
I20260924 15:18:22.675642 3964 raft_consensus.cc:2229] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:18:22.675725 3964 raft_consensus.cc:2258] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:22.675863 3963 tablet_replica.cc:330] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b: stopping tablet replica
I20260924 15:18:22.675943 3963 raft_consensus.cc:2229] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:18:22.676029 3963 raft_consensus.cc:2258] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:22.676214 3964 ts_tablet_manager.cc:1905] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:22.676448 3963 ts_tablet_manager.cc:1905] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:22.678133 3964 ts_tablet_manager.cc:1918] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:18:22.678196 3964 log.cc:1196] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/c6ee64d0bd0947adbd80bbfd0ce07b2c
I20260924 15:18:22.678182 3963 ts_tablet_manager.cc:1918] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:18:22.678244 3963 log.cc:1196] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/c6ee64d0bd0947adbd80bbfd0ce07b2c
I20260924 15:18:22.678479 3964 ts_tablet_manager.cc:1939] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 27358c8ff08b41fbbd7980ccd096b0bf: Deleting consensus metadata
I20260924 15:18:22.678506 3963 ts_tablet_manager.cc:1939] T c6ee64d0bd0947adbd80bbfd0ce07b2c P 03bb374658c8465481e991e78007a38b: Deleting consensus metadata
I20260924 15:18:22.678974 3452 catalog_manager.cc:4841] TS 27358c8ff08b41fbbd7980ccd096b0bf (127.31.12.67:34707): tablet c6ee64d0bd0947adbd80bbfd0ce07b2c (table test-table [id=4278f26e490642149fa4c8735fdeebec]) successfully deleted
I20260924 15:18:22.679248 3452 catalog_manager.cc:4841] TS 03bb374658c8465481e991e78007a38b (127.31.12.66:33413): tablet c6ee64d0bd0947adbd80bbfd0ce07b2c (table test-table [id=4278f26e490642149fa4c8735fdeebec]) successfully deleted
W20260924 15:18:22.682355 3453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.12.65:44927 (error 108)
I20260924 15:18:22.682848 3936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.682713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:44927 (local address 127.0.0.1:56774)
0924 15:18:22.682776 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:22.682816 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.65:44927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 15:18:22.682905 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 3663
W20260924 15:18:22.684623 3453 catalog_manager.cc:4584] TS 6f23ffe289a543518b472f6fd28d7ef3 (127.31.12.65:44927): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c6ee64d0bd0947adbd80bbfd0ce07b2c: Network error: Client connection negotiation failed: client connection to 127.31.12.65:44927: connect: Connection refused (error 111)
I20260924 15:18:22.689015 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 3797
I20260924 15:18:22.694902 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 3435
2026-09-24T15:18:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2151 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T15:18:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:18:22Z Disabled control of system clock
I20260924 15:18:22.724452 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:35519
--webserver_interface=127.31.12.126
--webserver_port=0
--builtin_ntp_servers=127.31.12.84:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:35519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 15:18:22.804064 3969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:22.804267 3969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:22.804291 3969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:22.805370 3969 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:18:22.805408 3969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:22.805420 3969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:22.805433 3969 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:18:22.805505 3969 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:18:22.806795 3969 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:35519
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:35519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:22.806989 3969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:22.807226 3969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:22.812117 3975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:22.812145 3977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:22.812147 3974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:22.812179 3969 server_base.cc:1034] running on GCE node
I20260924 15:18:22.812539 3969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:22.812780 3969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:22.813921 3969 hybrid_clock.cc:648] HybridClock initialized: now 1790263102813907 us; error 29 us; skew 500 ppm
I20260924 15:18:22.815163 3969 webserver.cc:460] Webserver started at http://127.31.12.126:39013/ using document root <none> and password file <none>
I20260924 15:18:22.815351 3969 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:22.815392 3969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:22.815492 3969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:22.816498 3969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "0d752cdf159d45eabb3c0b6b10a4871f"
format_stamp: "Formatted at 2026-09-24 15:18:22 on dist-test-slave-zwbb"
server_key: "48640df540883e86a81f5685b255d319"
server_key_iv: "61008b16f68efabf6654b684c5e0770e"
server_key_version: "encryptionkey@0"
I20260924 15:18:22.816797 3969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "0d752cdf159d45eabb3c0b6b10a4871f"
format_stamp: "Formatted at 2026-09-24 15:18:22 on dist-test-slave-zwbb"
server_key: "48640df540883e86a81f5685b255d319"
server_key_iv: "61008b16f68efabf6654b684c5e0770e"
server_key_version: "encryptionkey@0"
I20260924 15:18:22.818092 3969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:18:22.818807 3983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:22.819003 3969 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:22.819077 3969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "0d752cdf159d45eabb3c0b6b10a4871f"
format_stamp: "Formatted at 2026-09-24 15:18:22 on dist-test-slave-zwbb"
server_key: "48640df540883e86a81f5685b255d319"
server_key_iv: "61008b16f68efabf6654b684c5e0770e"
server_key_version: "encryptionkey@0"
I20260924 15:18:22.819131 3969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:22.822404 3969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:22.822639 3969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:22.822736 3969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:22.826611 3969 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:35519
I20260924 15:18:22.826663 4036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:35519 every 8 connection(s)
I20260924 15:18:22.826977 3969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:18:22.827657 4037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:22.828357 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 3969
I20260924 15:18:22.828477 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:18:22.828686 31793 external_mini_cluster.cc:1428] Setting key 624e27df6aa214ac82357caf987ff933
I20260924 15:18:22.831277 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.829278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58222 (local address 127.31.12.126:35519)
0924 15:18:22.829520 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:22.829527 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:22.829557 (+ 30us) server_negotiation.cc:407] Connection header received
0924 15:18:22.829982 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:22.829988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:22.830028 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:22.830141 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:22.830805 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:22.830808 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:22.830808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:18:22.830812 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:22.830854 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:18:22.830864 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:22.830917 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:22.831019 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 15:18:22.831094 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260924 15:18:22.831828 4037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f: Bootstrap starting.
I20260924 15:18:22.832432 4037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:22.832715 4037 log.cc:824] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:22.833379 4037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f: No bootstrap required, opened a new log
I20260924 15:18:22.834702 4037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d752cdf159d45eabb3c0b6b10a4871f" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35519 } }
I20260924 15:18:22.834860 4037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:22.834913 4037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d752cdf159d45eabb3c0b6b10a4871f, State: Initialized, Role: FOLLOWER
I20260924 15:18:22.835032 4037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d752cdf159d45eabb3c0b6b10a4871f" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35519 } }
I20260924 15:18:22.835140 4037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:18:22.835186 4037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:18:22.835238 4037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:22.836071 4037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d752cdf159d45eabb3c0b6b10a4871f" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35519 } }
I20260924 15:18:22.836190 4037 leader_election.cc:302] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d752cdf159d45eabb3c0b6b10a4871f; no voters:
I20260924 15:18:22.836361 4037 leader_election.cc:288] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:18:22.836405 4042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:22.836611 4042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f [term 1 LEADER]: Becoming Leader. State: Replica: 0d752cdf159d45eabb3c0b6b10a4871f, State: Running, Role: LEADER
I20260924 15:18:22.836622 4037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:18:22.836727 4042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d752cdf159d45eabb3c0b6b10a4871f" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35519 } }
I20260924 15:18:22.837108 4043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d752cdf159d45eabb3c0b6b10a4871f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d752cdf159d45eabb3c0b6b10a4871f" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35519 } } }
I20260924 15:18:22.837221 4043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:22.837103 4044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d752cdf159d45eabb3c0b6b10a4871f. Latest consensus state: current_term: 1 leader_uuid: "0d752cdf159d45eabb3c0b6b10a4871f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d752cdf159d45eabb3c0b6b10a4871f" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35519 } } }
I20260924 15:18:22.837380 4044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:22.838057 4051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:18:22.838585 4051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:18:22.840133 4051 catalog_manager.cc:1347] Generated new cluster ID: fb41607190804a8f84bc4a0d381a1384
I20260924 15:18:22.840212 4051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:18:22.857487 4051 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:18:22.857686 4051 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:18:22.865051 4051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f: Generated new TSK 0
I20260924 15:18:22.865253 4051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:18:22.867760 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:35519
--builtin_ntp_servers=127.31.12.84:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 15:18:22.951121 4061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:22.951333 4061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:22.951356 4061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:22.952608 4061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:22.952646 4061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:22.952673 4061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:18:22.954123 4061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:35519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:22.954378 4061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:22.954691 4061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:22.959652 4066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:22.959645 4067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:22.959878 4061 server_base.cc:1034] running on GCE node
W20260924 15:18:22.959916 4069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:22.960191 4061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:22.960479 4061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:22.961632 4061 hybrid_clock.cc:648] HybridClock initialized: now 1790263102961612 us; error 37 us; skew 500 ppm
I20260924 15:18:22.962954 4061 webserver.cc:460] Webserver started at http://127.31.12.65:37969/ using document root <none> and password file <none>
I20260924 15:18:22.963239 4061 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:22.963284 4061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:22.963434 4061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:22.964486 4061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "b7dc327e003a4e5c91563b3b9b02b881"
format_stamp: "Formatted at 2026-09-24 15:18:22 on dist-test-slave-zwbb"
server_key: "3b883d6ea5bb227055be6ade7b603826"
server_key_iv: "e4274254995cd4654038fc55155b1516"
server_key_version: "encryptionkey@0"
I20260924 15:18:22.964841 4061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "b7dc327e003a4e5c91563b3b9b02b881"
format_stamp: "Formatted at 2026-09-24 15:18:22 on dist-test-slave-zwbb"
server_key: "3b883d6ea5bb227055be6ade7b603826"
server_key_iv: "e4274254995cd4654038fc55155b1516"
server_key_version: "encryptionkey@0"
I20260924 15:18:22.966260 4061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:18:22.966965 4075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:22.967167 4061 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:18:22.967257 4061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "b7dc327e003a4e5c91563b3b9b02b881"
format_stamp: "Formatted at 2026-09-24 15:18:22 on dist-test-slave-zwbb"
server_key: "3b883d6ea5bb227055be6ade7b603826"
server_key_iv: "e4274254995cd4654038fc55155b1516"
server_key_version: "encryptionkey@0"
I20260924 15:18:22.967329 4061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:22.970659 4061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:22.970916 4061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:22.971051 4061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:22.971253 4061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:22.971544 4061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:22.971575 4061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:22.971625 4061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:22.971652 4061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:22.977299 4061 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:34931
I20260924 15:18:22.977344 4189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:34931 every 8 connection(s)
I20260924 15:18:22.977680 4061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:18:22.980715 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.978387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:53507 (local address 127.31.12.126:35519)
0924 15:18:22.978479 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:22.978481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:22.978680 (+ 199us) server_negotiation.cc:407] Connection header received
0924 15:18:22.979197 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:22.979199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:22.979206 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:22.979261 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:22.980455 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:22.980456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:22.980457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:18:22.980462 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:22.980478 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:18:22.980482 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:22.980534 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:22.980625 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 15:18:22.980644 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 15:18:22.980904 4192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:22.978370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:35519 (local address 127.31.12.65:53507)
0924 15:18:22.978564 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:22.978598 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:22.979089 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:22.979340 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:22.979344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:22.979354 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:22.980301 (+ 947us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:18:22.980301 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:22.980329 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:18:22.980356 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:22.980532 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:22.980536 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:22.980540 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:18:22.980624 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 15:18:22.980689 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260924 15:18:22.981383 4190 heartbeater.cc:347] Connected to a master server at 127.31.12.126:35519
I20260924 15:18:22.981530 4190 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:22.981668 4190 heartbeater.cc:511] Master 127.31.12.126:35519 requested a full tablet report, sending...
I20260924 15:18:22.982139 4000 ts_manager.cc:194] Registered new tserver with Master: b7dc327e003a4e5c91563b3b9b02b881 (127.31.12.65:34931)
I20260924 15:18:22.982681 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 4061
I20260924 15:18:22.982760 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:18:22.982983 31793 external_mini_cluster.cc:1428] Setting key 11a217448f91085a7f9440f4514a120c
I20260924 15:18:22.983848 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:35519
--builtin_ntp_servers=127.31.12.84:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 15:18:23.063937 4193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:23.064177 4193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:23.064213 4193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:23.065644 4193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:23.065698 4193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:23.065744 4193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:18:23.067052 4193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:35519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:23.067236 4193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:23.067468 4193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:23.072459 4198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:23.072459 4201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:23.072487 4193 server_base.cc:1034] running on GCE node
W20260924 15:18:23.072481 4199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:23.072922 4193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:23.073238 4193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:23.074399 4193 hybrid_clock.cc:648] HybridClock initialized: now 1790263103074385 us; error 32 us; skew 500 ppm
I20260924 15:18:23.075565 4193 webserver.cc:460] Webserver started at http://127.31.12.66:44245/ using document root <none> and password file <none>
I20260924 15:18:23.075771 4193 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:23.075814 4193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:23.075902 4193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:23.076920 4193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "a0d21aed24ac4de4abac142026b1d6ad"
format_stamp: "Formatted at 2026-09-24 15:18:23 on dist-test-slave-zwbb"
server_key: "42282c70fa5173e3e95630f5ae590776"
server_key_iv: "2f6fd1895d02eb34f6e669a2f4702b92"
server_key_version: "encryptionkey@0"
I20260924 15:18:23.077227 4193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "a0d21aed24ac4de4abac142026b1d6ad"
format_stamp: "Formatted at 2026-09-24 15:18:23 on dist-test-slave-zwbb"
server_key: "42282c70fa5173e3e95630f5ae590776"
server_key_iv: "2f6fd1895d02eb34f6e669a2f4702b92"
server_key_version: "encryptionkey@0"
I20260924 15:18:23.078516 4193 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:18:23.079172 4207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:23.079327 4193 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:23.079396 4193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "a0d21aed24ac4de4abac142026b1d6ad"
format_stamp: "Formatted at 2026-09-24 15:18:23 on dist-test-slave-zwbb"
server_key: "42282c70fa5173e3e95630f5ae590776"
server_key_iv: "2f6fd1895d02eb34f6e669a2f4702b92"
server_key_version: "encryptionkey@0"
I20260924 15:18:23.079447 4193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:23.082818 4193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:23.083051 4193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:23.083145 4193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:23.083292 4193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:23.083527 4193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:23.083556 4193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:23.083576 4193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:23.083590 4193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:23.089143 4193 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:33381
I20260924 15:18:23.089186 4321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:33381 every 8 connection(s)
I20260924 15:18:23.089547 4193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:18:23.092291 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.090238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:39457 (local address 127.31.12.126:35519)
0924 15:18:23.090297 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:23.090299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:23.090509 (+ 210us) server_negotiation.cc:407] Connection header received
0924 15:18:23.090932 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:23.090933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:23.090937 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:23.090998 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:23.092128 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:23.092129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:23.092129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:18:23.092132 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:23.092142 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:18:23.092145 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:23.092183 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:23.092229 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 15:18:23.092239 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:18:23.092540 4324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.090199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:35519 (local address 127.31.12.66:39457)
0924 15:18:23.090373 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:23.090413 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:23.090854 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:23.091047 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:23.091050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:23.091058 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:23.092006 (+ 948us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:18:23.092007 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:23.092034 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:18:23.092060 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:23.092183 (+ 123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:23.092185 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:23.092188 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:18:23.092236 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 15:18:23.092282 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260924 15:18:23.093065 4322 heartbeater.cc:347] Connected to a master server at 127.31.12.126:35519
I20260924 15:18:23.093163 4322 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:23.093300 4322 heartbeater.cc:511] Master 127.31.12.126:35519 requested a full tablet report, sending...
I20260924 15:18:23.093727 4000 ts_manager.cc:194] Registered new tserver with Master: a0d21aed24ac4de4abac142026b1d6ad (127.31.12.66:33381)
I20260924 15:18:23.098354 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 4193
I20260924 15:18:23.098486 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:18:23.098668 31793 external_mini_cluster.cc:1428] Setting key 6802065ad07b59c9c37c1adf84732d5c
I20260924 15:18:23.099546 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:35519
--builtin_ntp_servers=127.31.12.84:34885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 15:18:23.179533 4325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:23.179724 4325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:23.179746 4325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:23.180897 4325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:23.180936 4325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:23.180962 4325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:18:23.182875 4325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:35519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:23.183125 4325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:23.183436 4325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:23.188329 4331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:23.188377 4325 server_base.cc:1034] running on GCE node
W20260924 15:18:23.188329 4333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:23.188385 4330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:23.188747 4325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:23.189016 4325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:23.190174 4325 hybrid_clock.cc:648] HybridClock initialized: now 1790263103190173 us; error 39 us; skew 500 ppm
I20260924 15:18:23.191380 4325 webserver.cc:460] Webserver started at http://127.31.12.67:38083/ using document root <none> and password file <none>
I20260924 15:18:23.191649 4325 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:23.191720 4325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:23.191870 4325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:23.192905 4325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "a55af579e7c44ae2a8c7f4f6678169e9"
format_stamp: "Formatted at 2026-09-24 15:18:23 on dist-test-slave-zwbb"
server_key: "04a690c8e6a5f449d304d1749c05edac"
server_key_iv: "e1b94feca760aebd654028e6cce99ee3"
server_key_version: "encryptionkey@0"
I20260924 15:18:23.193264 4325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "a55af579e7c44ae2a8c7f4f6678169e9"
format_stamp: "Formatted at 2026-09-24 15:18:23 on dist-test-slave-zwbb"
server_key: "04a690c8e6a5f449d304d1749c05edac"
server_key_iv: "e1b94feca760aebd654028e6cce99ee3"
server_key_version: "encryptionkey@0"
I20260924 15:18:23.194689 4325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:18:23.195402 4339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:23.195574 4325 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:18:23.195673 4325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "a55af579e7c44ae2a8c7f4f6678169e9"
format_stamp: "Formatted at 2026-09-24 15:18:23 on dist-test-slave-zwbb"
server_key: "04a690c8e6a5f449d304d1749c05edac"
server_key_iv: "e1b94feca760aebd654028e6cce99ee3"
server_key_version: "encryptionkey@0"
I20260924 15:18:23.195751 4325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:23.199090 4325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:23.199354 4325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:23.199497 4325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:23.199705 4325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:23.200006 4325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:23.200064 4325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:23.200104 4325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:23.200132 4325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:23.205549 4325 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:33015
I20260924 15:18:23.205605 4453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:33015 every 8 connection(s)
I20260924 15:18:23.205909 4325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:18:23.208647 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.206565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:57445 (local address 127.31.12.126:35519)
0924 15:18:23.206604 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:23.206607 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:23.206808 (+ 201us) server_negotiation.cc:407] Connection header received
0924 15:18:23.207297 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:23.207298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:23.207306 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:23.207357 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:23.208424 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:23.208426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:23.208426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:18:23.208431 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:23.208444 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:18:23.208448 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:23.208499 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:23.208564 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 15:18:23.208577 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:18:23.208887 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.206537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:35519 (local address 127.31.12.67:57445)
0924 15:18:23.206696 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:23.206726 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:23.207180 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:23.207420 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:23.207424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:23.207434 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:23.208280 (+ 846us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:18:23.208281 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:23.208306 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:18:23.208332 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:23.208496 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:23.208500 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:23.208504 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:18:23.208576 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 15:18:23.208634 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":67,"threads_started":1}
I20260924 15:18:23.209331 4454 heartbeater.cc:347] Connected to a master server at 127.31.12.126:35519
I20260924 15:18:23.209414 4454 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:23.209599 4454 heartbeater.cc:511] Master 127.31.12.126:35519 requested a full tablet report, sending...
I20260924 15:18:23.209934 4000 ts_manager.cc:194] Registered new tserver with Master: a55af579e7c44ae2a8c7f4f6678169e9 (127.31.12.67:33015)
I20260924 15:18:23.214108 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 4325
I20260924 15:18:23.214187 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:18:23.214313 31793 external_mini_cluster.cc:1428] Setting key 2e8cbae2cc8fde63f92efb5eb62fc786
I20260924 15:18:23.215119 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:18:23.217315 4040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.216334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.31.12.126:35519)
0924 15:18:23.216375 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:23.216377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:23.216495 (+ 118us) server_negotiation.cc:407] Connection header received
0924 15:18:23.216549 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:23.216550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:23.216554 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:23.216597 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:23.217140 (+ 543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:23.217141 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:23.217141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:18:23.217144 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:23.217153 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:18:23.217156 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:23.217191 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:23.217224 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 15:18:23.217234 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:18:23.218890 4000 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:18:23.220360 4000 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:18:23.223978 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.222415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38948 (local address 127.31.12.66:33381)
0924 15:18:23.222615 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:23.222620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:23.222629 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:18:23.222654 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:23.222656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:23.222665 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:23.222746 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:23.223598 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:23.223609 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:23.223610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:18:23.223614 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:23.223651 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:18:23.223678 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:23.223717 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:23.223871 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 15:18:23.223890 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":79,"threads_started":1}
I20260924 15:18:23.224223 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.222326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:33381 (local address 127.0.0.1:38948)
0924 15:18:23.222483 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:23.222521 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:23.222576 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:23.222952 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:23.222959 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:23.222965 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:23.223461 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:18:23.223461 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:23.223479 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:18:23.223501 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:23.223766 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:23.223772 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:23.223775 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:18:23.224111 (+ 336us) client_negotiation.cc:240] Negotiation successful
0924 15:18:23.224139 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":56,"threads_started":1}
I20260924 15:18:23.224222 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.222332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:33015 (local address 127.0.0.1:52084)
0924 15:18:23.222654 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:23.222666 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:23.222723 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:23.223397 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:23.223398 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:23.223401 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:23.223801 (+ 400us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:18:23.223801 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:23.223808 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:18:23.223820 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:23.224083 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:23.224084 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:23.224085 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 15:18:23.224114 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 15:18:23.224145 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"mutex_wait_us":18,"thread_start_us":78,"threads_started":1}
I20260924 15:18:23.224371 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.222399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52084 (local address 127.31.12.67:33015)
0924 15:18:23.223240 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:23.223245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:23.223253 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:18:23.223273 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:23.223275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:23.223283 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:23.223569 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:23.223955 (+ 386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:23.223958 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:23.223958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:18:23.223961 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:23.223988 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:18:23.224001 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:23.224244 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:23.224267 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 15:18:23.224282 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":75,"threads_started":1}
I20260924 15:18:23.224388 4463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.222525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:34931 (local address 127.0.0.1:44506)
0924 15:18:23.223068 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:23.223078 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:23.223132 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:23.223649 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:23.223650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:23.223653 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:23.223987 (+ 334us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:18:23.223988 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:23.223995 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:18:23.224006 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:23.224255 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:23.224256 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:23.224256 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 15:18:23.224300 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 15:18:23.224314 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":86,"threads_started":1}
I20260924 15:18:23.224584 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.222776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44506 (local address 127.31.12.65:34931)
0924 15:18:23.222956 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:23.222959 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:23.223212 (+ 253us) server_negotiation.cc:407] Connection header received
0924 15:18:23.223244 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:23.223246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:23.223256 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:23.223326 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:23.224105 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:23.224114 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:23.224115 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:18:23.224119 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:23.224153 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:18:23.224181 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:23.224230 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:23.224467 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 15:18:23.224486 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":97,"threads_started":1}
I20260924 15:18:23.224972 4256 tablet_service.cc:1467] Processing CreateTablet for tablet 3170f145b56a4dfca01b9acd3dfb1ab9 (DEFAULT_TABLE table=test-table [id=ed6131d9272b466098094ec6e11871eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:23.225335 4256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3170f145b56a4dfca01b9acd3dfb1ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:23.225376 4123 tablet_service.cc:1467] Processing CreateTablet for tablet 3170f145b56a4dfca01b9acd3dfb1ab9 (DEFAULT_TABLE table=test-table [id=ed6131d9272b466098094ec6e11871eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:23.225423 4388 tablet_service.cc:1467] Processing CreateTablet for tablet 3170f145b56a4dfca01b9acd3dfb1ab9 (DEFAULT_TABLE table=test-table [id=ed6131d9272b466098094ec6e11871eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:23.225740 4388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3170f145b56a4dfca01b9acd3dfb1ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:23.225767 4123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3170f145b56a4dfca01b9acd3dfb1ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:23.228231 4468 tablet_bootstrap.cc:492] T 3170f145b56a4dfca01b9acd3dfb1ab9 P b7dc327e003a4e5c91563b3b9b02b881: Bootstrap starting.
I20260924 15:18:23.228641 4469 tablet_bootstrap.cc:492] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad: Bootstrap starting.
I20260924 15:18:23.229005 4470 tablet_bootstrap.cc:492] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9: Bootstrap starting.
I20260924 15:18:23.229161 4468 tablet_bootstrap.cc:654] T 3170f145b56a4dfca01b9acd3dfb1ab9 P b7dc327e003a4e5c91563b3b9b02b881: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:23.229265 4469 tablet_bootstrap.cc:654] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:23.229555 4468 log.cc:824] T 3170f145b56a4dfca01b9acd3dfb1ab9 P b7dc327e003a4e5c91563b3b9b02b881: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:23.229610 4469 log.cc:824] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:23.229668 4470 tablet_bootstrap.cc:654] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:23.229979 4470 log.cc:824] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:23.230376 4468 tablet_bootstrap.cc:492] T 3170f145b56a4dfca01b9acd3dfb1ab9 P b7dc327e003a4e5c91563b3b9b02b881: No bootstrap required, opened a new log
I20260924 15:18:23.230451 4468 ts_tablet_manager.cc:1397] T 3170f145b56a4dfca01b9acd3dfb1ab9 P b7dc327e003a4e5c91563b3b9b02b881: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 15:18:23.230517 4469 tablet_bootstrap.cc:492] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad: No bootstrap required, opened a new log
I20260924 15:18:23.230711 4470 tablet_bootstrap.cc:492] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9: No bootstrap required, opened a new log
I20260924 15:18:23.230773 4470 ts_tablet_manager.cc:1397] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:18:23.230764 4469 ts_tablet_manager.cc:1397] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:18:23.232331 4468 raft_consensus.cc:348] T 3170f145b56a4dfca01b9acd3dfb1ab9 P b7dc327e003a4e5c91563b3b9b02b881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d21aed24ac4de4abac142026b1d6ad" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33381 } } peers { permanent_uuid: "a55af579e7c44ae2a8c7f4f6678169e9" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33015 } } peers { permanent_uuid: "b7dc327e003a4e5c91563b3b9b02b881" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34931 } }
I20260924 15:18:23.232483 4468 raft_consensus.cc:374] T 3170f145b56a4dfca01b9acd3dfb1ab9 P b7dc327e003a4e5c91563b3b9b02b881 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:23.232527 4468 raft_consensus.cc:729] T 3170f145b56a4dfca01b9acd3dfb1ab9 P b7dc327e003a4e5c91563b3b9b02b881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7dc327e003a4e5c91563b3b9b02b881, State: Initialized, Role: FOLLOWER
I20260924 15:18:23.232651 4468 consensus_queue.cc:260] T 3170f145b56a4dfca01b9acd3dfb1ab9 P b7dc327e003a4e5c91563b3b9b02b881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d21aed24ac4de4abac142026b1d6ad" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33381 } } peers { permanent_uuid: "a55af579e7c44ae2a8c7f4f6678169e9" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33015 } } peers { permanent_uuid: "b7dc327e003a4e5c91563b3b9b02b881" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34931 } }
I20260924 15:18:23.232913 4468 ts_tablet_manager.cc:1428] T 3170f145b56a4dfca01b9acd3dfb1ab9 P b7dc327e003a4e5c91563b3b9b02b881: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 15:18:23.233067 4190 heartbeater.cc:503] Master 127.31.12.126:35519 was elected leader, sending a full tablet report...
I20260924 15:18:23.233353 4470 raft_consensus.cc:348] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d21aed24ac4de4abac142026b1d6ad" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33381 } } peers { permanent_uuid: "a55af579e7c44ae2a8c7f4f6678169e9" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33015 } } peers { permanent_uuid: "b7dc327e003a4e5c91563b3b9b02b881" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34931 } }
I20260924 15:18:23.233530 4470 raft_consensus.cc:374] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:23.233569 4470 raft_consensus.cc:729] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a55af579e7c44ae2a8c7f4f6678169e9, State: Initialized, Role: FOLLOWER
I20260924 15:18:23.233353 4469 raft_consensus.cc:348] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d21aed24ac4de4abac142026b1d6ad" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33381 } } peers { permanent_uuid: "a55af579e7c44ae2a8c7f4f6678169e9" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33015 } } peers { permanent_uuid: "b7dc327e003a4e5c91563b3b9b02b881" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34931 } }
I20260924 15:18:23.233605 4469 raft_consensus.cc:374] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:23.233639 4469 raft_consensus.cc:729] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0d21aed24ac4de4abac142026b1d6ad, State: Initialized, Role: FOLLOWER
I20260924 15:18:23.233671 4470 consensus_queue.cc:260] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d21aed24ac4de4abac142026b1d6ad" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33381 } } peers { permanent_uuid: "a55af579e7c44ae2a8c7f4f6678169e9" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33015 } } peers { permanent_uuid: "b7dc327e003a4e5c91563b3b9b02b881" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34931 } }
I20260924 15:18:23.233748 4469 consensus_queue.cc:260] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d21aed24ac4de4abac142026b1d6ad" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33381 } } peers { permanent_uuid: "a55af579e7c44ae2a8c7f4f6678169e9" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33015 } } peers { permanent_uuid: "b7dc327e003a4e5c91563b3b9b02b881" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34931 } }
I20260924 15:18:23.233912 4470 ts_tablet_manager.cc:1428] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:18:23.233992 4469 ts_tablet_manager.cc:1428] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 15:18:23.234019 4454 heartbeater.cc:503] Master 127.31.12.126:35519 was elected leader, sending a full tablet report...
I20260924 15:18:23.234040 4322 heartbeater.cc:503] Master 127.31.12.126:35519 was elected leader, sending a full tablet report...
W20260924 15:18:23.340050 4323 tablet.cc:2369] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:18:23.363276 4475 raft_consensus.cc:482] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:18:23.363399 4475 raft_consensus.cc:504] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d21aed24ac4de4abac142026b1d6ad" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33381 } } peers { permanent_uuid: "a55af579e7c44ae2a8c7f4f6678169e9" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33015 } } peers { permanent_uuid: "b7dc327e003a4e5c91563b3b9b02b881" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34931 } }
I20260924 15:18:23.363797 4475 leader_election.cc:288] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0d21aed24ac4de4abac142026b1d6ad (127.31.12.66:33381), b7dc327e003a4e5c91563b3b9b02b881 (127.31.12.65:34931)
I20260924 15:18:23.365206 4456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.363984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:33381 (local address 127.31.12.67:50701)
0924 15:18:23.364090 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:23.364102 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:23.364148 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:23.364310 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:23.364311 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:23.364313 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:23.364796 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:18:23.364797 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:23.364805 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:18:23.364819 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:23.365029 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:23.365030 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:23.365031 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 15:18:23.365104 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 15:18:23.365119 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 15:18:23.365234 4465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.364078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:50701 (local address 127.31.12.66:33381)
0924 15:18:23.364141 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:23.364144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:23.364157 (+ 13us) server_negotiation.cc:407] Connection header received
0924 15:18:23.364231 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:23.364232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:23.364238 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:23.364294 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:23.364901 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:23.364903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:23.364903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:18:23.364907 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:23.364936 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:18:23.364956 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:23.365023 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:23.365117 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 15:18:23.365137 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 15:18:23.365338 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.364099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:39439 (local address 127.31.12.65:34931)
0924 15:18:23.364161 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:23.364163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:23.364265 (+ 102us) server_negotiation.cc:407] Connection header received
0924 15:18:23.364312 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:23.364313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:23.364319 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:23.364377 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:23.365106 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:23.365108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:23.365108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:18:23.365113 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:23.365129 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:18:23.365133 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:23.365186 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:23.365244 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 15:18:23.365258 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:18:23.365348 4478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.364009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:34931 (local address 127.31.12.67:39439)
0924 15:18:23.364176 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:23.364188 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:23.364252 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:23.364392 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:23.364393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:23.364395 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:23.364999 (+ 604us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:18:23.365000 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:23.365008 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:18:23.365020 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:23.365193 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:23.365194 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:23.365195 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 15:18:23.365245 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 15:18:23.365258 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
I20260924 15:18:23.365533 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3170f145b56a4dfca01b9acd3dfb1ab9" candidate_uuid: "a55af579e7c44ae2a8c7f4f6678169e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d21aed24ac4de4abac142026b1d6ad" is_pre_election: true
I20260924 15:18:23.365661 4276 raft_consensus.cc:2454] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a55af579e7c44ae2a8c7f4f6678169e9 in term 0.
I20260924 15:18:23.365713 4144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3170f145b56a4dfca01b9acd3dfb1ab9" candidate_uuid: "a55af579e7c44ae2a8c7f4f6678169e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7dc327e003a4e5c91563b3b9b02b881" is_pre_election: true
I20260924 15:18:23.365866 4144 raft_consensus.cc:2454] T 3170f145b56a4dfca01b9acd3dfb1ab9 P b7dc327e003a4e5c91563b3b9b02b881 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a55af579e7c44ae2a8c7f4f6678169e9 in term 0.
I20260924 15:18:23.365882 4341 leader_election.cc:302] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d21aed24ac4de4abac142026b1d6ad, a55af579e7c44ae2a8c7f4f6678169e9; no voters:
I20260924 15:18:23.366014 4475 raft_consensus.cc:2781] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:18:23.366096 4475 raft_consensus.cc:482] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:18:23.366127 4475 raft_consensus.cc:3037] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:23.366832 4475 raft_consensus.cc:504] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d21aed24ac4de4abac142026b1d6ad" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33381 } } peers { permanent_uuid: "a55af579e7c44ae2a8c7f4f6678169e9" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33015 } } peers { permanent_uuid: "b7dc327e003a4e5c91563b3b9b02b881" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34931 } }
I20260924 15:18:23.366978 4475 leader_election.cc:288] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [CANDIDATE]: Term 1 election: Requested vote from peers a0d21aed24ac4de4abac142026b1d6ad (127.31.12.66:33381), b7dc327e003a4e5c91563b3b9b02b881 (127.31.12.65:34931)
I20260924 15:18:23.367118 4276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3170f145b56a4dfca01b9acd3dfb1ab9" candidate_uuid: "a55af579e7c44ae2a8c7f4f6678169e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0d21aed24ac4de4abac142026b1d6ad"
I20260924 15:18:23.367146 4144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3170f145b56a4dfca01b9acd3dfb1ab9" candidate_uuid: "a55af579e7c44ae2a8c7f4f6678169e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7dc327e003a4e5c91563b3b9b02b881"
I20260924 15:18:23.367197 4276 raft_consensus.cc:3037] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:23.367228 4144 raft_consensus.cc:3037] T 3170f145b56a4dfca01b9acd3dfb1ab9 P b7dc327e003a4e5c91563b3b9b02b881 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:23.367913 4276 raft_consensus.cc:2454] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a55af579e7c44ae2a8c7f4f6678169e9 in term 1.
I20260924 15:18:23.367937 4144 raft_consensus.cc:2454] T 3170f145b56a4dfca01b9acd3dfb1ab9 P b7dc327e003a4e5c91563b3b9b02b881 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a55af579e7c44ae2a8c7f4f6678169e9 in term 1.
I20260924 15:18:23.368093 4341 leader_election.cc:302] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0d21aed24ac4de4abac142026b1d6ad, a55af579e7c44ae2a8c7f4f6678169e9; no voters:
I20260924 15:18:23.368181 4475 raft_consensus.cc:2781] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:23.368345 4475 raft_consensus.cc:686] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [term 1 LEADER]: Becoming Leader. State: Replica: a55af579e7c44ae2a8c7f4f6678169e9, State: Running, Role: LEADER
I20260924 15:18:23.368454 4475 consensus_queue.cc:237] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d21aed24ac4de4abac142026b1d6ad" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33381 } } peers { permanent_uuid: "a55af579e7c44ae2a8c7f4f6678169e9" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33015 } } peers { permanent_uuid: "b7dc327e003a4e5c91563b3b9b02b881" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34931 } }
I20260924 15:18:23.369338 4000 catalog_manager.cc:5495] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a55af579e7c44ae2a8c7f4f6678169e9 (127.31.12.67). New cstate: current_term: 1 leader_uuid: "a55af579e7c44ae2a8c7f4f6678169e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0d21aed24ac4de4abac142026b1d6ad" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a55af579e7c44ae2a8c7f4f6678169e9" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dc327e003a4e5c91563b3b9b02b881" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34931 } health_report { overall_health: UNKNOWN } } }
I20260924 15:18:23.408247 4464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.407156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52092 (local address 127.31.12.67:33015)
0924 15:18:23.407226 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:23.407229 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:23.407239 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:18:23.407317 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:23.407318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:23.407332 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:23.407399 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:23.408003 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:23.408004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:23.408005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:18:23.408009 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:23.408025 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:18:23.408029 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:23.408103 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:23.408152 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 15:18:23.408168 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:18:23.410041 4276 raft_consensus.cc:1261] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad [term 1 FOLLOWER]: Refusing update from remote peer a55af579e7c44ae2a8c7f4f6678169e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:23.410066 4144 raft_consensus.cc:1261] T 3170f145b56a4dfca01b9acd3dfb1ab9 P b7dc327e003a4e5c91563b3b9b02b881 [term 1 FOLLOWER]: Refusing update from remote peer a55af579e7c44ae2a8c7f4f6678169e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:23.410388 4479 consensus_queue.cc:1035] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0d21aed24ac4de4abac142026b1d6ad" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:23.410530 4475 consensus_queue.cc:1035] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dc327e003a4e5c91563b3b9b02b881" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:23.412794 4482 mvcc.cc:205] Tried to move back new op lower bound from 7332917671564959744 to 7332917671398039552. Current Snapshot: MvccSnapshot[applied={T|T < 7332917671564959744}]
I20260924 15:18:23.413359 4484 mvcc.cc:205] Tried to move back new op lower bound from 7332917671564959744 to 7332917671398039552. Current Snapshot: MvccSnapshot[applied={T|T < 7332917671564959744}]
I20260924 15:18:23.415208 3998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:18:23.415309 3998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:18:23.416105 3998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d752cdf159d45eabb3c0b6b10a4871f: Sending DeleteTablet for 3 replicas of tablet 3170f145b56a4dfca01b9acd3dfb1ab9
I20260924 15:18:23.416457 4256 tablet_service.cc:1514] Processing DeleteTablet for tablet 3170f145b56a4dfca01b9acd3dfb1ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:23 UTC) from {username='slave'} at 127.0.0.1:38948
I20260924 15:18:23.416476 4388 tablet_service.cc:1514] Processing DeleteTablet for tablet 3170f145b56a4dfca01b9acd3dfb1ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:23 UTC) from {username='slave'} at 127.0.0.1:52084
I20260924 15:18:23.416535 4123 tablet_service.cc:1514] Processing DeleteTablet for tablet 3170f145b56a4dfca01b9acd3dfb1ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:23 UTC) from {username='slave'} at 127.0.0.1:44506
I20260924 15:18:23.416724 4493 tablet_replica.cc:330] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad: stopping tablet replica
I20260924 15:18:23.416783 4494 tablet_replica.cc:330] T 3170f145b56a4dfca01b9acd3dfb1ab9 P b7dc327e003a4e5c91563b3b9b02b881: stopping tablet replica
I20260924 15:18:23.416880 4494 raft_consensus.cc:2229] T 3170f145b56a4dfca01b9acd3dfb1ab9 P b7dc327e003a4e5c91563b3b9b02b881 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:18:23.416903 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 4061
I20260924 15:18:23.416991 4494 raft_consensus.cc:2258] T 3170f145b56a4dfca01b9acd3dfb1ab9 P b7dc327e003a4e5c91563b3b9b02b881 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:23.416904 4493 raft_consensus.cc:2229] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:18:23.417936 4493 raft_consensus.cc:2258] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:23.418462 4493 ts_tablet_manager.cc:1905] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:23.418494 4492 tablet_replica.cc:330] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9: stopping tablet replica
I20260924 15:18:23.418581 4492 raft_consensus.cc:2229] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:18:23.418709 4492 raft_consensus.cc:2258] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:23.420542 4493 ts_tablet_manager.cc:1918] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:18:23.420677 4493 log.cc:1196] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/3170f145b56a4dfca01b9acd3dfb1ab9
I20260924 15:18:23.421043 4493 ts_tablet_manager.cc:1939] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a0d21aed24ac4de4abac142026b1d6ad: Deleting consensus metadata
I20260924 15:18:23.421653 3985 catalog_manager.cc:4841] TS a0d21aed24ac4de4abac142026b1d6ad (127.31.12.66:33381): tablet 3170f145b56a4dfca01b9acd3dfb1ab9 (table test-table [id=ed6131d9272b466098094ec6e11871eb]) successfully deleted
I20260924 15:18:23.422178 4492 ts_tablet_manager.cc:1905] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:23.423918 4492 ts_tablet_manager.cc:1918] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:18:23.423986 4492 log.cc:1196] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/3170f145b56a4dfca01b9acd3dfb1ab9
I20260924 15:18:23.424265 4492 ts_tablet_manager.cc:1939] T 3170f145b56a4dfca01b9acd3dfb1ab9 P a55af579e7c44ae2a8c7f4f6678169e9: Deleting consensus metadata
I20260924 15:18:23.424700 3987 catalog_manager.cc:4841] TS a55af579e7c44ae2a8c7f4f6678169e9 (127.31.12.67:33015): tablet 3170f145b56a4dfca01b9acd3dfb1ab9 (table test-table [id=ed6131d9272b466098094ec6e11871eb]) successfully deleted
W20260924 15:18:23.427812 3987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.12.65:34931 (error 108)
I20260924 15:18:23.428287 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.428170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:34931 (local address 127.0.0.1:44516)
0924 15:18:23.428220 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:23.428259 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.65:34931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 15:18:23.428414 3987 catalog_manager.cc:4584] TS b7dc327e003a4e5c91563b3b9b02b881 (127.31.12.65:34931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3170f145b56a4dfca01b9acd3dfb1ab9: Network error: Client connection negotiation failed: client connection to 127.31.12.65:34931: connect: Connection refused (error 111)
I20260924 15:18:23.428701 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 4193
I20260924 15:18:23.433779 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 4325
I20260924 15:18:23.438584 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 3969
2026-09-24T15:18:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (741 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 15:18:23.458434 31793 mini_oidc.cc:152] Starting JWKS server
I20260924 15:18:23.459529 31793 webserver.cc:460] Webserver started at https://127.0.0.1:36035/ using document root <none> and password file <none>
I20260924 15:18:23.459656 31793 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 15:18:23.459846 31793 webserver.cc:460] Webserver started at http://127.0.0.1:37285/ using document root <none> and password file <none>
2026-09-24T15:18:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:18:23Z Disabled control of system clock
I20260924 15:18:23.468602 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:40403
--webserver_interface=127.31.12.126
--webserver_port=0
--builtin_ntp_servers=127.31.12.84:39959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:40403
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36035/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263080303576-31793-0/testchainca.cert with env {}
W20260924 15:18:23.550302 4501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:23.550514 4501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:23.550536 4501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:23.551632 4501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:18:23.551666 4501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:23.551679 4501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:23.551692 4501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:18:23.551703 4501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 15:18:23.551716 4501 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 15:18:23.551734 4501 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36035/jwks.json
I20260924 15:18:23.553200 4501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:39959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:40403
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:40403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36035/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263080303576-31793-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:23.553392 4501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:23.553646 4501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:23.558388 4506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:23.558396 4509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:23.558396 4507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:23.559139 4501 server_base.cc:1034] running on GCE node
I20260924 15:18:23.559386 4501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:23.559629 4501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:23.560771 4501 hybrid_clock.cc:648] HybridClock initialized: now 1790263103560758 us; error 29 us; skew 500 ppm
I20260924 15:18:23.562204 4501 webserver.cc:460] Webserver started at http://127.31.12.126:34139/ using document root <none> and password file <none>
I20260924 15:18:23.562461 4501 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:23.562508 4501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:23.562659 4501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:23.563733 4501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "fcc67fbadbf24e6ea759c834c4e821a1"
format_stamp: "Formatted at 2026-09-24 15:18:23 on dist-test-slave-zwbb"
server_key: "fc41eeac8e1f928558b97150e2d742b7"
server_key_iv: "cd195cc39b70e781aa196daba59a50b2"
server_key_version: "encryptionkey@0"
I20260924 15:18:23.564108 4501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "fcc67fbadbf24e6ea759c834c4e821a1"
format_stamp: "Formatted at 2026-09-24 15:18:23 on dist-test-slave-zwbb"
server_key: "fc41eeac8e1f928558b97150e2d742b7"
server_key_iv: "cd195cc39b70e781aa196daba59a50b2"
server_key_version: "encryptionkey@0"
I20260924 15:18:23.565508 4501 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:18:23.566512 4515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:23.566695 4501 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:23.566799 4501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "fcc67fbadbf24e6ea759c834c4e821a1"
format_stamp: "Formatted at 2026-09-24 15:18:23 on dist-test-slave-zwbb"
server_key: "fc41eeac8e1f928558b97150e2d742b7"
server_key_iv: "cd195cc39b70e781aa196daba59a50b2"
server_key_version: "encryptionkey@0"
I20260924 15:18:23.566902 4501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:23.580691 4501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:23.581022 4501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:23.581180 4501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:23.585238 4501 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:40403
I20260924 15:18:23.585273 4567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:40403 every 8 connection(s)
I20260924 15:18:23.585685 4501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:18:23.586652 4568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:23.588853 4568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1: Bootstrap starting.
I20260924 15:18:23.589491 4568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:23.589785 4568 log.cc:824] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:23.590451 4568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1: No bootstrap required, opened a new log
I20260924 15:18:23.591997 4568 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc67fbadbf24e6ea759c834c4e821a1" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 40403 } }
I20260924 15:18:23.592130 4568 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:23.592178 4568 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcc67fbadbf24e6ea759c834c4e821a1, State: Initialized, Role: FOLLOWER
I20260924 15:18:23.592341 4568 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc67fbadbf24e6ea759c834c4e821a1" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 40403 } }
I20260924 15:18:23.592422 4568 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:18:23.592482 4568 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:18:23.592535 4568 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:23.593042 4568 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc67fbadbf24e6ea759c834c4e821a1" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 40403 } }
I20260924 15:18:23.593150 4568 leader_election.cc:302] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcc67fbadbf24e6ea759c834c4e821a1; no voters:
I20260924 15:18:23.593327 4568 leader_election.cc:288] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:18:23.593358 4571 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:23.593528 4571 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1 [term 1 LEADER]: Becoming Leader. State: Replica: fcc67fbadbf24e6ea759c834c4e821a1, State: Running, Role: LEADER
I20260924 15:18:23.593606 4568 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:18:23.593643 4571 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc67fbadbf24e6ea759c834c4e821a1" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 40403 } }
I20260924 15:18:23.593880 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 4501
I20260924 15:18:23.593986 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:18:23.594113 4572 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcc67fbadbf24e6ea759c834c4e821a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc67fbadbf24e6ea759c834c4e821a1" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 40403 } } }
I20260924 15:18:23.594182 31793 external_mini_cluster.cc:1428] Setting key d66bc486a435b8af72935b7ac8fd689d
I20260924 15:18:23.594218 4572 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1 [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:23.594216 4573 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcc67fbadbf24e6ea759c834c4e821a1. Latest consensus state: current_term: 1 leader_uuid: "fcc67fbadbf24e6ea759c834c4e821a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc67fbadbf24e6ea759c834c4e821a1" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 40403 } } }
I20260924 15:18:23.594445 4573 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1 [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:23.595036 4575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:18:23.595553 4575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:18:23.597056 4575 catalog_manager.cc:1347] Generated new cluster ID: 8ee2e31c0230494da949def88c301452
I20260924 15:18:23.597106 4575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:18:23.600454 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.594815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43834 (local address 127.31.12.126:40403)
0924 15:18:23.595063 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:23.595072 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:23.595137 (+ 65us) server_negotiation.cc:407] Connection header received
0924 15:18:23.595249 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:23.595257 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:23.595335 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:23.595473 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:23.595816 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:23.597198 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:23.598309 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:23.598907 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:23.599031 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:23.599745 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:23.599748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:23.599749 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:18:23.599963 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:23.600001 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:18:23.600014 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:23.600073 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:23.600169 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 15:18:23.600240 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":76,"threads_started":1}
I20260924 15:18:23.601339 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:18:23.614482 4575 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:18:23.615015 4575 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:18:23.620913 4575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcc67fbadbf24e6ea759c834c4e821a1: Generated new TSK 0
I20260924 15:18:23.621114 4575 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 15:18:23.646164 4599 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 15:18:23.646277 4599 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260924 15:18:23.646903 4598 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:18:23.630333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:40403 (local address 127.0.0.1:43842)
0924 15:18:23.630512 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:23.630521 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:23.630577 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:23.630754 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:23.630755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:23.630757 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 15:18:23.630805 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:23.630807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:23.631719 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:23.631720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:23.632300 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:23.632302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:23.632778 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:23.632778 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:23.632816 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:23.632835 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 15:18:23.646466 (+ 13631us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 15:18:23.646574 (+ 108us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.12.126:40403: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":93,"threads_started":1}
I20260924 15:18:23.647222 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.630419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43842 (local address 127.31.12.126:40403)
0924 15:18:23.630467 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:23.630469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:23.630586 (+ 117us) server_negotiation.cc:407] Connection header received
0924 15:18:23.630652 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:23.630653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:23.630665 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:23.630742 (+ 77us) server_negotiation.cc:226] Negotiated authn=JWT
0924 15:18:23.630875 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:23.631640 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:23.632385 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:23.632692 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:23.632763 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:23.632946 (+ 183us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 15:18:23.646309 (+ 13363us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 15:18:23.647157 (+ 848us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43842: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 15:18:23.647270 4583 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43842: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.31.12.126:40403: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36035/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 15:18:23.647398 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 4501
2026-09-24T15:18:23Z chronyd exiting
I20260924 15:18:23.683868 31793 test_util.cc:175] -----------------------------------------------
I20260924 15:18:23.683969 31793 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790263080303576-31793-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (226 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 15:18:23.684711 31793 mini_oidc.cc:152] Starting JWKS server
I20260924 15:18:23.685418 31793 webserver.cc:460] Webserver started at https://127.0.0.1:38969/ using document root <none> and password file <none>
I20260924 15:18:23.685518 31793 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 15:18:23.685704 31793 webserver.cc:460] Webserver started at http://127.0.0.1:37453/ using document root <none> and password file <none>
2026-09-24T15:18:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:18:23Z Disabled control of system clock
I20260924 15:18:23.693673 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:35605
--webserver_interface=127.31.12.126
--webserver_port=0
--builtin_ntp_servers=127.31.12.84:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:35605
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38969/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263080303576-31793-0/testchainca.cert with env {}
W20260924 15:18:23.777670 4605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:23.777855 4605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:23.777876 4605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:23.779023 4605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:18:23.779062 4605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:23.779075 4605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:23.779112 4605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:18:23.779138 4605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 15:18:23.779160 4605 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 15:18:23.779188 4605 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38969/jwks.json
I20260924 15:18:23.780676 4605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:35605
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:35605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38969/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263080303576-31793-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4605
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:23.780910 4605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:23.781117 4605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:23.785799 4613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:23.785823 4610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:23.785853 4605 server_base.cc:1034] running on GCE node
W20260924 15:18:23.785825 4611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:23.786201 4605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:23.786473 4605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:23.787623 4605 hybrid_clock.cc:648] HybridClock initialized: now 1790263103787615 us; error 33 us; skew 500 ppm
I20260924 15:18:23.788857 4605 webserver.cc:460] Webserver started at http://127.31.12.126:40481/ using document root <none> and password file <none>
I20260924 15:18:23.789112 4605 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:23.789157 4605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:23.789316 4605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:23.790370 4605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "06f6f00dd02145158340a8ffc69d710e"
format_stamp: "Formatted at 2026-09-24 15:18:23 on dist-test-slave-zwbb"
server_key: "c66085f4b299b58a96902d9db3c57b7a"
server_key_iv: "349cb2a67331beb38d603e250044e083"
server_key_version: "encryptionkey@0"
I20260924 15:18:23.790719 4605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "06f6f00dd02145158340a8ffc69d710e"
format_stamp: "Formatted at 2026-09-24 15:18:23 on dist-test-slave-zwbb"
server_key: "c66085f4b299b58a96902d9db3c57b7a"
server_key_iv: "349cb2a67331beb38d603e250044e083"
server_key_version: "encryptionkey@0"
I20260924 15:18:23.792011 4605 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:18:23.792781 4619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:23.792974 4605 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:23.793052 4605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "06f6f00dd02145158340a8ffc69d710e"
format_stamp: "Formatted at 2026-09-24 15:18:23 on dist-test-slave-zwbb"
server_key: "c66085f4b299b58a96902d9db3c57b7a"
server_key_iv: "349cb2a67331beb38d603e250044e083"
server_key_version: "encryptionkey@0"
I20260924 15:18:23.793107 4605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:23.811513 4605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:23.811779 4605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:23.811882 4605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:23.815611 4605 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:35605
I20260924 15:18:23.815665 4671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:35605 every 8 connection(s)
I20260924 15:18:23.816007 4605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:18:23.816694 4672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:23.819365 4672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e: Bootstrap starting.
I20260924 15:18:23.819849 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 4605
I20260924 15:18:23.819970 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:18:23.820037 4672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:23.820173 31793 external_mini_cluster.cc:1428] Setting key ec4aafde98b39fa0bcba07b799ef5150
I20260924 15:18:23.820392 4672 log.cc:824] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:23.821204 4672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e: No bootstrap required, opened a new log
I20260924 15:18:23.823765 4672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06f6f00dd02145158340a8ffc69d710e" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35605 } }
I20260924 15:18:23.823895 4672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:23.823917 4672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06f6f00dd02145158340a8ffc69d710e, State: Initialized, Role: FOLLOWER
I20260924 15:18:23.824047 4672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06f6f00dd02145158340a8ffc69d710e" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35605 } }
I20260924 15:18:23.824120 4672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:18:23.824143 4672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:18:23.824209 4672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:23.824924 4672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06f6f00dd02145158340a8ffc69d710e" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35605 } }
I20260924 15:18:23.825034 4672 leader_election.cc:302] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06f6f00dd02145158340a8ffc69d710e; no voters:
I20260924 15:18:23.825202 4672 leader_election.cc:288] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:18:23.825304 4677 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:23.825434 4672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:18:23.825860 4677 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e [term 1 LEADER]: Becoming Leader. State: Replica: 06f6f00dd02145158340a8ffc69d710e, State: Running, Role: LEADER
I20260924 15:18:23.825970 4677 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06f6f00dd02145158340a8ffc69d710e" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35605 } }
I20260924 15:18:23.826361 4679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06f6f00dd02145158340a8ffc69d710e. Latest consensus state: current_term: 1 leader_uuid: "06f6f00dd02145158340a8ffc69d710e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06f6f00dd02145158340a8ffc69d710e" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35605 } } }
I20260924 15:18:23.826458 4679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:23.826700 4678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06f6f00dd02145158340a8ffc69d710e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06f6f00dd02145158340a8ffc69d710e" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35605 } } }
I20260924 15:18:23.827148 4678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:23.826853 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.820693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59832 (local address 127.31.12.126:35605)
0924 15:18:23.820892 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:23.820901 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:23.820934 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:18:23.821619 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:23.821628 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:23.821704 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:23.821852 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:23.822057 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:23.823446 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:23.824392 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:23.825045 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:23.825177 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:23.825991 (+ 814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:23.825995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:23.825995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:18:23.826199 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:23.826249 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:18:23.826263 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:23.826330 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:23.826525 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 15:18:23.826613 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":66,"threads_started":1}
I20260924 15:18:23.826848 4689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:18:23.827661 4689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:18:23.828228 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:18:23.829121 4689 catalog_manager.cc:1347] Generated new cluster ID: 7c9be2b61b3f44cab6addee00dbd8527
I20260924 15:18:23.829172 4689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:18:23.849254 4689 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:18:23.849817 4689 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:18:23.854926 4689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06f6f00dd02145158340a8ffc69d710e: Generated new TSK 0
I20260924 15:18:23.855127 4689 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 15:18:23.886646 4702 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 15:18:23.886734 4702 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 15:18:23.887039 4676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:23.865072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59834 (local address 127.31.12.126:35605)
0924 15:18:23.865138 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:23.865140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:23.865262 (+ 122us) server_negotiation.cc:407] Connection header received
0924 15:18:23.865335 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:23.865336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:23.865346 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:23.865406 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0924 15:18:23.865600 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:23.866068 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:23.867210 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:23.867393 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:23.867440 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:23.867650 (+ 210us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 15:18:23.886844 (+ 19194us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 15:18:23.886984 (+ 140us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59834: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 15:18:23.887111 4676 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59834: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 15:18:23.887194 4701 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:18:23.865000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:35605 (local address 127.0.0.1:59834)
0924 15:18:23.865185 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:23.865199 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:23.865267 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:23.865455 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:23.865456 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:23.865459 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0924 15:18:23.865532 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:23.865534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:23.866164 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:23.866165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:23.867125 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:23.867129 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:23.867474 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:23.867476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:23.867541 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:23.867547 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 15:18:23.886966 (+ 19419us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 15:18:23.887095 (+ 129us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.12.126:35605: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38969/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":102,"threads_started":1}
I20260924 15:18:23.887494 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 4605
2026-09-24T15:18:23Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (240 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 15:18:23.925917 31793 mini_oidc.cc:152] Starting JWKS server
I20260924 15:18:23.926635 31793 webserver.cc:460] Webserver started at https://127.0.0.1:41113/ using document root <none> and password file <none>
I20260924 15:18:23.926710 31793 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 15:18:23.926896 31793 webserver.cc:460] Webserver started at http://127.0.0.1:33393/ using document root <none> and password file <none>
2026-09-24T15:18:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:18:23Z Disabled control of system clock
I20260924 15:18:23.934686 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:33759
--webserver_interface=127.31.12.126
--webserver_port=0
--builtin_ntp_servers=127.31.12.84:38431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:33759
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41113/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 15:18:24.014902 4708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:24.015101 4708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:24.015123 4708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:24.016294 4708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:18:24.016336 4708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:24.016356 4708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:24.016369 4708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:18:24.016381 4708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 15:18:24.016393 4708 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 15:18:24.016405 4708 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41113/jwks.json
I20260924 15:18:24.017848 4708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:38431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:33759
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:33759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41113/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:24.018098 4708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:24.018409 4708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:24.023165 4715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:24.023167 4717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:24.023188 4714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:24.023393 4708 server_base.cc:1034] running on GCE node
I20260924 15:18:24.023614 4708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:24.023855 4708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:24.025005 4708 hybrid_clock.cc:648] HybridClock initialized: now 1790263104024983 us; error 36 us; skew 500 ppm
I20260924 15:18:24.026233 4708 webserver.cc:460] Webserver started at http://127.31.12.126:44409/ using document root <none> and password file <none>
I20260924 15:18:24.026502 4708 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:24.026551 4708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:24.026695 4708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:24.027817 4708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "890e0594152548eabfa8ae395e8b4349"
format_stamp: "Formatted at 2026-09-24 15:18:24 on dist-test-slave-zwbb"
server_key: "0ea8887b543eae38091b9265887eebe6"
server_key_iv: "7ec91330149b87e317a98f8906c5566d"
server_key_version: "encryptionkey@0"
I20260924 15:18:24.028192 4708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "890e0594152548eabfa8ae395e8b4349"
format_stamp: "Formatted at 2026-09-24 15:18:24 on dist-test-slave-zwbb"
server_key: "0ea8887b543eae38091b9265887eebe6"
server_key_iv: "7ec91330149b87e317a98f8906c5566d"
server_key_version: "encryptionkey@0"
I20260924 15:18:24.029554 4708 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:18:24.030294 4723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:24.030460 4708 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:24.030550 4708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "890e0594152548eabfa8ae395e8b4349"
format_stamp: "Formatted at 2026-09-24 15:18:24 on dist-test-slave-zwbb"
server_key: "0ea8887b543eae38091b9265887eebe6"
server_key_iv: "7ec91330149b87e317a98f8906c5566d"
server_key_version: "encryptionkey@0"
I20260924 15:18:24.030630 4708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:24.052542 4708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:24.052891 4708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:24.053051 4708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:24.057163 4708 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:33759
I20260924 15:18:24.057197 4775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:33759 every 8 connection(s)
I20260924 15:18:24.057596 4708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:18:24.058255 4776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:24.058885 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 4708
I20260924 15:18:24.059007 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:18:24.059183 31793 external_mini_cluster.cc:1428] Setting key 2482a2517e1484122331b84fa254c1cc
I20260924 15:18:24.061271 4776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349: Bootstrap starting.
I20260924 15:18:24.061977 4776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:24.062279 4776 log.cc:824] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:24.063114 4776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349: No bootstrap required, opened a new log
I20260924 15:18:24.065115 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:24.059717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.31.12.126:33759)
0924 15:18:24.059904 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:24.059911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:24.059936 (+ 25us) server_negotiation.cc:407] Connection header received
0924 15:18:24.060324 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:24.060330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:24.060393 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:24.060498 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:24.060749 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:24.061731 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:24.062872 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:24.063567 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:24.063672 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:24.064406 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:24.064411 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:24.064411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:18:24.064578 (+ 167us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:24.064609 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:18:24.064620 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:24.064674 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:24.064789 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 15:18:24.064906 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
I20260924 15:18:24.065414 4776 raft_consensus.cc:348] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890e0594152548eabfa8ae395e8b4349" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33759 } }
I20260924 15:18:24.065622 4776 raft_consensus.cc:374] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:24.065672 4776 raft_consensus.cc:729] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 890e0594152548eabfa8ae395e8b4349, State: Initialized, Role: FOLLOWER
I20260924 15:18:24.065801 4776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890e0594152548eabfa8ae395e8b4349" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33759 } }
I20260924 15:18:24.065891 4776 raft_consensus.cc:388] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:18:24.065958 4776 raft_consensus.cc:482] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:18:24.066008 4776 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:24.066100 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:18:24.066567 4776 raft_consensus.cc:504] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890e0594152548eabfa8ae395e8b4349" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33759 } }
I20260924 15:18:24.066684 4776 leader_election.cc:302] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 890e0594152548eabfa8ae395e8b4349; no voters:
I20260924 15:18:24.066849 4776 leader_election.cc:288] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:18:24.066907 4781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:24.067057 4781 raft_consensus.cc:686] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349 [term 1 LEADER]: Becoming Leader. State: Replica: 890e0594152548eabfa8ae395e8b4349, State: Running, Role: LEADER
I20260924 15:18:24.067080 4776 sys_catalog.cc:564] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:18:24.067149 4781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890e0594152548eabfa8ae395e8b4349" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33759 } }
I20260924 15:18:24.067476 4781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 890e0594152548eabfa8ae395e8b4349. Latest consensus state: current_term: 1 leader_uuid: "890e0594152548eabfa8ae395e8b4349" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890e0594152548eabfa8ae395e8b4349" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33759 } } }
I20260924 15:18:24.067552 4781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349 [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:24.067513 4783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "890e0594152548eabfa8ae395e8b4349" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "890e0594152548eabfa8ae395e8b4349" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33759 } } }
I20260924 15:18:24.067600 4783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349 [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:24.067763 4788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:18:24.068313 4788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:18:24.069808 4788 catalog_manager.cc:1347] Generated new cluster ID: fca704cca9bc4bb387f90c462f801a4d
I20260924 15:18:24.069860 4788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:18:24.084805 4788 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:18:24.085330 4788 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:18:24.088305 4788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 890e0594152548eabfa8ae395e8b4349: Generated new TSK 0
I20260924 15:18:24.088474 4788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:18:24.099148 4779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:24.095797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60058 (local address 127.31.12.126:33759)
0924 15:18:24.095860 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:24.095862 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:24.095949 (+ 87us) server_negotiation.cc:407] Connection header received
0924 15:18:24.096035 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:24.096035 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:24.096046 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:24.096132 (+ 86us) server_negotiation.cc:226] Negotiated authn=JWT
0924 15:18:24.096242 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:24.096756 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:24.097619 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:24.097836 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:24.097941 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:24.098112 (+ 171us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 15:18:24.098904 (+ 792us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41113/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 15:18:24.099067 (+ 163us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60058: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41113/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":40}
W20260924 15:18:24.099216 4779 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60058: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41113/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 15:18:24.099215 4805 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:18:24.095731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33759 (local address 127.0.0.1:60058)
0924 15:18:24.095901 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:24.095910 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:24.095968 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:24.096131 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:24.096132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:24.096134 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 15:18:24.096187 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:24.096190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:24.096842 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:24.096843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:24.097532 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:24.097536 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:24.097932 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:24.097933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:24.097996 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:24.098001 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 15:18:24.099018 (+ 1017us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41113/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 15:18:24.099110 (+ 92us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.12.126:33759: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41113/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":100,"threads_started":1}
I20260924 15:18:24.099570 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 4708
E20260924 15:18:24.110446 4806 webserver.cc:533] Webserver: sslize failed:
E20260924 15:18:24.110505 4806 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T15:18:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (211 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T15:18:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:18:24Z Disabled control of system clock
I20260924 15:18:24.144964 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:43919
--webserver_interface=127.31.12.126
--webserver_port=0
--builtin_ntp_servers=127.31.12.84:41263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:43919
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263080303576-31793-0/testchainca.cert with env {}
W20260924 15:18:24.224946 4810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:24.225143 4810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:24.225164 4810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:24.226413 4810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:18:24.226459 4810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:24.226471 4810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:24.226485 4810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:18:24.226496 4810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 15:18:24.226548 4810 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 15:18:24.226565 4810 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 15:18:24.228046 4810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:41263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:43919
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:43919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263080303576-31793-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:24.228294 4810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:24.228551 4810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:24.233475 4818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:24.233492 4816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:24.233525 4815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:24.233521 4810 server_base.cc:1034] running on GCE node
I20260924 15:18:24.233811 4810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:24.234118 4810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:24.235291 4810 hybrid_clock.cc:648] HybridClock initialized: now 1790263104235273 us; error 47 us; skew 500 ppm
I20260924 15:18:24.236601 4810 webserver.cc:460] Webserver started at http://127.31.12.126:42665/ using document root <none> and password file <none>
I20260924 15:18:24.236861 4810 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:24.236938 4810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:24.237089 4810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:24.238225 4810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "910289ccb64f451c86be92825986c0cf"
format_stamp: "Formatted at 2026-09-24 15:18:24 on dist-test-slave-zwbb"
server_key: "ac7673fe31de302f53a01edfb09ca3cd"
server_key_iv: "348bd8392c8aa37b00299c093b84226c"
server_key_version: "encryptionkey@0"
I20260924 15:18:24.238590 4810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "910289ccb64f451c86be92825986c0cf"
format_stamp: "Formatted at 2026-09-24 15:18:24 on dist-test-slave-zwbb"
server_key: "ac7673fe31de302f53a01edfb09ca3cd"
server_key_iv: "348bd8392c8aa37b00299c093b84226c"
server_key_version: "encryptionkey@0"
I20260924 15:18:24.239915 4810 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:18:24.240625 4824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:24.240799 4810 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:18:24.240900 4810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "910289ccb64f451c86be92825986c0cf"
format_stamp: "Formatted at 2026-09-24 15:18:24 on dist-test-slave-zwbb"
server_key: "ac7673fe31de302f53a01edfb09ca3cd"
server_key_iv: "348bd8392c8aa37b00299c093b84226c"
server_key_version: "encryptionkey@0"
I20260924 15:18:24.240984 4810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:24.247804 4810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:24.248099 4810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:24.248257 4810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:24.252118 4810 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:43919
I20260924 15:18:24.252148 4877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:43919 every 8 connection(s)
I20260924 15:18:24.252485 4810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:18:24.253415 4878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:24.255669 4878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf: Bootstrap starting.
I20260924 15:18:24.256191 4878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:24.256417 4878 log.cc:824] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:24.257040 4878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf: No bootstrap required, opened a new log
I20260924 15:18:24.258677 4878 raft_consensus.cc:348] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910289ccb64f451c86be92825986c0cf" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43919 } }
I20260924 15:18:24.258810 4878 raft_consensus.cc:374] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:24.258832 4878 raft_consensus.cc:729] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 910289ccb64f451c86be92825986c0cf, State: Initialized, Role: FOLLOWER
I20260924 15:18:24.258966 4878 consensus_queue.cc:260] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910289ccb64f451c86be92825986c0cf" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43919 } }
I20260924 15:18:24.259052 4878 raft_consensus.cc:388] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:18:24.259076 4878 raft_consensus.cc:482] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:18:24.259130 4878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:24.259128 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 4810
I20260924 15:18:24.259256 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:18:24.259454 31793 external_mini_cluster.cc:1428] Setting key 865c59d41bf41a05798a34f59ab689e7
I20260924 15:18:24.259815 4878 raft_consensus.cc:504] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910289ccb64f451c86be92825986c0cf" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43919 } }
I20260924 15:18:24.259935 4878 leader_election.cc:302] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 910289ccb64f451c86be92825986c0cf; no voters:
I20260924 15:18:24.260102 4878 leader_election.cc:288] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:18:24.260130 4883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:24.260278 4883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf [term 1 LEADER]: Becoming Leader. State: Replica: 910289ccb64f451c86be92825986c0cf, State: Running, Role: LEADER
I20260924 15:18:24.260368 4883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910289ccb64f451c86be92825986c0cf" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43919 } }
I20260924 15:18:24.260465 4878 sys_catalog.cc:564] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:18:24.260663 4883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 910289ccb64f451c86be92825986c0cf. Latest consensus state: current_term: 1 leader_uuid: "910289ccb64f451c86be92825986c0cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910289ccb64f451c86be92825986c0cf" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43919 } } }
I20260924 15:18:24.260752 4883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:24.260676 4884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "910289ccb64f451c86be92825986c0cf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "910289ccb64f451c86be92825986c0cf" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43919 } } }
I20260924 15:18:24.260871 4884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:24.261057 4887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:18:24.261706 4887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:18:24.263710 4894 mvcc.cc:205] Tried to move back new op lower bound from 7332917675062222848 to 7332917675050815488. Current Snapshot: MvccSnapshot[applied={T|T < 7332917675062222848}]
I20260924 15:18:24.264088 4887 catalog_manager.cc:1347] Generated new cluster ID: 220799c9e59c41699cd19fbc89b94c03
I20260924 15:18:24.264153 4887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:18:24.266259 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:24.259940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40438 (local address 127.31.12.126:43919)
0924 15:18:24.260274 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:24.260282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:24.260317 (+ 35us) server_negotiation.cc:407] Connection header received
0924 15:18:24.260433 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:24.260439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:24.260513 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:24.260944 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:24.261033 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:24.263105 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:24.264230 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:24.264686 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:24.264835 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:24.265601 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:24.265604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:24.265605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:18:24.265779 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:24.265811 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:18:24.265822 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:24.265911 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:24.265980 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 15:18:24.266062 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":90,"threads_started":1}
I20260924 15:18:24.267215 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:18:24.284720 4887 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:18:24.285364 4887 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:18:24.291002 4887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 910289ccb64f451c86be92825986c0cf: Generated new TSK 0
I20260924 15:18:24.291195 4887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:18:24.315138 4882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:24.299925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40446 (local address 127.31.12.126:43919)
0924 15:18:24.299986 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:24.299988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:24.300097 (+ 109us) server_negotiation.cc:407] Connection header received
0924 15:18:24.300152 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:24.300152 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:24.300164 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:24.300242 (+ 78us) server_negotiation.cc:226] Negotiated authn=JWT
0924 15:18:24.300348 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:24.301135 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:24.301937 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:24.302264 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:24.302335 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:24.302483 (+ 148us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 15:18:24.314872 (+ 12389us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 15:18:24.315059 (+ 187us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40446: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 15:18:24.315209 4882 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40446: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 15:18:24.315224 4908 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:18:24.299843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43919 (local address 127.0.0.1:40446)
0924 15:18:24.300024 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:24.300034 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:24.300096 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:24.300249 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:24.300250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:24.300252 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 15:18:24.300303 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:24.300305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:24.301212 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:24.301213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:24.301855 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:24.301857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:24.302345 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:24.302346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:24.302384 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:24.302389 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 15:18:24.315023 (+ 12634us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 15:18:24.315120 (+ 97us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.31.12.126:43919: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":118,"threads_started":1}
I20260924 15:18:24.315670 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 4810
2026-09-24T15:18:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (195 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (90 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (87 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:18:24 dist-test-slave-zwbb krb5kdc[4920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:18:24 dist-test-slave-zwbb krb5kdc[4920](info): set up 2 sockets
Sep 24 15:18:24 dist-test-slave-zwbb krb5kdc[4920](info): commencing operation
krb5kdc: starting...
W20260924 15:18:25.531598 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:18:25 dist-test-slave-zwbb krb5kdc[4920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:18:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:18:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:18:25.652318 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:45575
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:45575
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:25.732873 4933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:25.733091 4933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:25.733125 4933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:25.734311 4933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:18:25.734375 4933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:25.734409 4933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:25.734441 4933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:18:25.734467 4933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:18:25.735879 4933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:45575
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:45575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:25.736153 4933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:25.736372 4933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:25.741103 4938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:25.741109 4939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:25.741129 4941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:25.741552 4933 server_base.cc:1034] running on GCE node
I20260924 15:18:25.741813 4933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:25.742226 4933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:25.743436 4933 hybrid_clock.cc:648] HybridClock initialized: now 1790263105743388 us; error 37 us; skew 500 ppm
Sep 24 15:18:25 dist-test-slave-zwbb krb5kdc[4920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263105, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:25.750771 4933 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:18:25.751266 4933 webserver.cc:460] Webserver started at http://127.31.12.126:37401/ using document root <none> and password file <none>
I20260924 15:18:25.751493 4933 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:25.751561 4933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:25.751714 4933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:25.752746 4933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "68287a7322d545bda3ca3184e1afef2a"
format_stamp: "Formatted at 2026-09-24 15:18:25 on dist-test-slave-zwbb"
server_key: "3aaa6db83a26e86aa1d1a5b32f7c23a1"
server_key_iv: "91fd3aa0821e4c05b5f0f9efb65379ff"
server_key_version: "encryptionkey@0"
I20260924 15:18:25.753113 4933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "68287a7322d545bda3ca3184e1afef2a"
format_stamp: "Formatted at 2026-09-24 15:18:25 on dist-test-slave-zwbb"
server_key: "3aaa6db83a26e86aa1d1a5b32f7c23a1"
server_key_iv: "91fd3aa0821e4c05b5f0f9efb65379ff"
server_key_version: "encryptionkey@0"
I20260924 15:18:25.754541 4933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:18:25.755304 4948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:25.755475 4933 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:18:25.755578 4933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "68287a7322d545bda3ca3184e1afef2a"
format_stamp: "Formatted at 2026-09-24 15:18:25 on dist-test-slave-zwbb"
server_key: "3aaa6db83a26e86aa1d1a5b32f7c23a1"
server_key_iv: "91fd3aa0821e4c05b5f0f9efb65379ff"
server_key_version: "encryptionkey@0"
I20260924 15:18:25.755652 4933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:25.775350 4933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:25.777607 4933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:25.777774 4933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:25.781767 4933 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:45575
I20260924 15:18:25.781801 5001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:45575 every 8 connection(s)
I20260924 15:18:25.782137 4933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:18:25.782932 5002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:25.785133 5002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a: Bootstrap starting.
I20260924 15:18:25.785662 5002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:25.785897 5002 log.cc:824] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:25.786532 5002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a: No bootstrap required, opened a new log
I20260924 15:18:25.787292 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 4933
I20260924 15:18:25.787390 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:18:25.787586 31793 external_mini_cluster.cc:1428] Setting key 10804792100cc2408bfb8f990556098b
I20260924 15:18:25.788650 5002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68287a7322d545bda3ca3184e1afef2a" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 45575 } }
I20260924 15:18:25.788815 5002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:25.788854 5002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68287a7322d545bda3ca3184e1afef2a, State: Initialized, Role: FOLLOWER
I20260924 15:18:25.789016 5002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68287a7322d545bda3ca3184e1afef2a" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 45575 } }
I20260924 15:18:25.789121 5002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:18:25.789157 5002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:18:25.789201 5002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:25.790165 5002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68287a7322d545bda3ca3184e1afef2a" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 45575 } }
I20260924 15:18:25.790252 5002 leader_election.cc:302] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68287a7322d545bda3ca3184e1afef2a; no voters:
I20260924 15:18:25.790414 5002 leader_election.cc:288] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:18:25.790474 5007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:25.790608 5007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a [term 1 LEADER]: Becoming Leader. State: Replica: 68287a7322d545bda3ca3184e1afef2a, State: Running, Role: LEADER
I20260924 15:18:25.790678 5002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:18:25.790700 5007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68287a7322d545bda3ca3184e1afef2a" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 45575 } }
I20260924 15:18:25.791061 5008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68287a7322d545bda3ca3184e1afef2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68287a7322d545bda3ca3184e1afef2a" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 45575 } } }
I20260924 15:18:25.791152 5008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:25.791360 5009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68287a7322d545bda3ca3184e1afef2a. Latest consensus state: current_term: 1 leader_uuid: "68287a7322d545bda3ca3184e1afef2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68287a7322d545bda3ca3184e1afef2a" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 45575 } } }
I20260924 15:18:25.791445 5009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:25.791584 5014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:18:25.792081 5014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:18:25.793825 5014 catalog_manager.cc:1347] Generated new cluster ID: 7e1548c8b4fc4a73b93ce45953d1a12d
I20260924 15:18:25.793895 5014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:18:25 dist-test-slave-zwbb krb5kdc[4920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:25.799496 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:25.788092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34404 (local address 127.31.12.126:45575)
0924 15:18:25.788267 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:25.788274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:25.788298 (+ 24us) server_negotiation.cc:407] Connection header received
0924 15:18:25.788629 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:25.788635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:25.788686 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:25.788792 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:25.789614 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:25.790694 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:25.791850 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:25.792636 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:25.792776 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:25.795531 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:25.795535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:25.795536 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:25.795548 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:25.797566 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:25.797975 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:25.797978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:25.797978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:25.798106 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:25.798333 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:25.798334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:25.798335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:25.798665 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:25.798841 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:25.799127 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 15:18:25.799277 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260924 15:18:25.799715 5014 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:18:25.800280 5014 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:18:25.816198 5014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68287a7322d545bda3ca3184e1afef2a: Generated new TSK 0
I20260924 15:18:25.816413 5014 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
I20260924 15:18:25.848376 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:45575
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:25.949211 5028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:25.949402 5028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:25.949424 5028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:25.950588 5028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:25.950630 5028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:25.950678 5028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:18:25.952199 5028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:45575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:25.952445 5028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:25.952669 5028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:25.957641 5033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:25.957657 5036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:25.957697 5028 server_base.cc:1034] running on GCE node
W20260924 15:18:25.957669 5034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:25.958107 5028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:25.958346 5028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:25.959507 5028 hybrid_clock.cc:648] HybridClock initialized: now 1790263105959487 us; error 35 us; skew 500 ppm
Sep 24 15:18:25 dist-test-slave-zwbb krb5kdc[4920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263105, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:25.962044 5028 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:18:25.962451 5028 webserver.cc:460] Webserver started at http://127.31.12.65:41543/ using document root <none> and password file <none>
I20260924 15:18:25.962683 5028 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:25.962750 5028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:25.962899 5028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:25.963912 5028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "af47ed9db12946fc8319d8829c982fc0"
format_stamp: "Formatted at 2026-09-24 15:18:25 on dist-test-slave-zwbb"
server_key: "fbd1898b81a02b850ed827d51464aa96"
server_key_iv: "d6334b6d46c8aca541c3f9e4e06c2000"
server_key_version: "encryptionkey@0"
I20260924 15:18:25.964258 5028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "af47ed9db12946fc8319d8829c982fc0"
format_stamp: "Formatted at 2026-09-24 15:18:25 on dist-test-slave-zwbb"
server_key: "fbd1898b81a02b850ed827d51464aa96"
server_key_iv: "d6334b6d46c8aca541c3f9e4e06c2000"
server_key_version: "encryptionkey@0"
I20260924 15:18:25.965675 5028 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:18:25.966404 5043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:25.966588 5028 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:25.966692 5028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "af47ed9db12946fc8319d8829c982fc0"
format_stamp: "Formatted at 2026-09-24 15:18:25 on dist-test-slave-zwbb"
server_key: "fbd1898b81a02b850ed827d51464aa96"
server_key_iv: "d6334b6d46c8aca541c3f9e4e06c2000"
server_key_version: "encryptionkey@0"
I20260924 15:18:25.966763 5028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:25.988046 5028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:25.990003 5028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:25.990170 5028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:25.990403 5028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:25.990726 5028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:25.990795 5028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:25.990839 5028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:25.990868 5028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:25.996593 5028 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:37197
I20260924 15:18:25.996636 5156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:37197 every 8 connection(s)
I20260924 15:18:25.996992 5028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:18:26 dist-test-slave-zwbb krb5kdc[4920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263105, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:26.003995 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 5028
I20260924 15:18:26.004132 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:18:26.004361 31793 external_mini_cluster.cc:1428] Setting key d1fba3a1ab8a01af24f20dff3e4e80bc
I20260924 15:18:26.006768 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:25.997717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:60093 (local address 127.31.12.126:45575)
0924 15:18:25.997787 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:25.997789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:25.998010 (+ 221us) server_negotiation.cc:407] Connection header received
0924 15:18:25.998991 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:25.998992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:25.999004 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:25.999076 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:25.999814 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:26.000351 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:26.001347 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:26.001647 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:26.001732 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:26.003789 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:26.003790 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:26.003790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:26.003798 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:26.005331 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:26.005792 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:26.005793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:26.005794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:26.005853 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:26.006102 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:26.006103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:26.006104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:26.006219 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:26.006291 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:26.006572 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 15:18:26.006674 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:18:26.006950 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:25.997709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:45575 (local address 127.31.12.65:60093)
0924 15:18:25.997887 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:25.997923 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:25.998876 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:25.999153 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:25.999157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:25.999421 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:25.999706 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:25.999713 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:26.000453 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:26.000454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:26.001250 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:26.001252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:26.001747 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:26.001748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:26.001874 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:26.002322 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:26.002330 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:26.003684 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:26.005460 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:26.005468 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:26.005470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:26.005699 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:26.005922 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:26.005923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:26.005923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:26.006033 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:26.006296 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:26.006304 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:26.006437 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 15:18:26.006543 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 15:18:26.006700 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260924 15:18:26.007433 5159 heartbeater.cc:347] Connected to a master server at 127.31.12.126:45575
I20260924 15:18:26.007541 5159 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:26.007686 5159 heartbeater.cc:511] Master 127.31.12.126:45575 requested a full tablet report, sending...
I20260924 15:18:26.008188 4965 ts_manager.cc:194] Registered new tserver with Master: af47ed9db12946fc8319d8829c982fc0 (127.31.12.65:37197)
I20260924 15:18:26.009109 4965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:60093
WARNING: no policy specified for kudu/127.31.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
I20260924 15:18:26.028261 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:45575
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:26.107981 5164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:26.108166 5164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:26.108189 5164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:26.109313 5164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:26.109354 5164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:26.109380 5164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:18:26.110905 5164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:45575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:26.111090 5164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:26.111318 5164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:18:26.116360 5164 server_base.cc:1034] running on GCE node
W20260924 15:18:26.116376 5170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:26.116403 5172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:26.116376 5169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:26.116866 5164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:26.117120 5164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:26.118283 5164 hybrid_clock.cc:648] HybridClock initialized: now 1790263106118279 us; error 37 us; skew 500 ppm
Sep 24 15:18:26 dist-test-slave-zwbb krb5kdc[4920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263106, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:26.120790 5164 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:18:26.121220 5164 webserver.cc:460] Webserver started at http://127.31.12.66:36225/ using document root <none> and password file <none>
I20260924 15:18:26.121515 5164 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:26.121588 5164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:26.121749 5164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:26.122795 5164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "7a1f636c2fb24437946b39b5d741b6af"
format_stamp: "Formatted at 2026-09-24 15:18:26 on dist-test-slave-zwbb"
server_key: "7b0cd1929bf5ae8cfa7a98c587d2f992"
server_key_iv: "7fbd4adaa174989a99d8724c06df69ed"
server_key_version: "encryptionkey@0"
I20260924 15:18:26.123169 5164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "7a1f636c2fb24437946b39b5d741b6af"
format_stamp: "Formatted at 2026-09-24 15:18:26 on dist-test-slave-zwbb"
server_key: "7b0cd1929bf5ae8cfa7a98c587d2f992"
server_key_iv: "7fbd4adaa174989a99d8724c06df69ed"
server_key_version: "encryptionkey@0"
I20260924 15:18:26.124449 5164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:18:26.125231 5180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:26.125401 5164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:18:26.125528 5164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "7a1f636c2fb24437946b39b5d741b6af"
format_stamp: "Formatted at 2026-09-24 15:18:26 on dist-test-slave-zwbb"
server_key: "7b0cd1929bf5ae8cfa7a98c587d2f992"
server_key_iv: "7fbd4adaa174989a99d8724c06df69ed"
server_key_version: "encryptionkey@0"
I20260924 15:18:26.125608 5164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:26.136693 5164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:26.139058 5164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:26.139219 5164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:26.139448 5164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:26.139747 5164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:26.139801 5164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:26.139838 5164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:26.139866 5164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:26.145401 5164 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:46155
I20260924 15:18:26.145526 5296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:46155 every 8 connection(s)
I20260924 15:18:26.145830 5164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:18:26 dist-test-slave-zwbb krb5kdc[4920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263106, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:26.152918 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 5164
I20260924 15:18:26.153046 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:18:26.153239 31793 external_mini_cluster.cc:1428] Setting key 5126fbb8b1df84a6d050b2efadf8d3b8
I20260924 15:18:26.155025 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:26.146570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:50501 (local address 127.31.12.126:45575)
0924 15:18:26.146638 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:26.146640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:26.146833 (+ 193us) server_negotiation.cc:407] Connection header received
0924 15:18:26.147734 (+ 901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:26.147735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:26.147745 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:26.147804 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:26.148344 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:26.148840 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:26.149848 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:26.150039 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:26.150093 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:26.152112 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:26.152113 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:26.152114 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:26.152120 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:26.153552 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:26.154015 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:26.154016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:26.154016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:26.154094 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:26.154355 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:26.154356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:26.154356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:26.154476 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:26.154551 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:26.154848 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 15:18:26.154937 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:18:26.155470 5299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:26.146556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:45575 (local address 127.31.12.66:50501)
0924 15:18:26.146709 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:26.146750 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:26.147650 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:26.147858 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:26.147861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:26.148102 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:26.148278 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:26.148282 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:26.148918 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:26.148919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:26.149778 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:26.149781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:26.150111 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:26.150112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:26.150206 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:26.150707 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:26.150714 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:26.152019 (+ 1305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:26.153612 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:26.153619 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:26.153621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:26.153914 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:26.154173 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:26.154174 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:26.154175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:26.154287 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:26.154572 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:26.154579 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:26.154712 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 15:18:26.154814 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 15:18:26.154968 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260924 15:18:26.155967 5297 heartbeater.cc:347] Connected to a master server at 127.31.12.126:45575
I20260924 15:18:26.156071 5297 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:26.156260 5297 heartbeater.cc:511] Master 127.31.12.126:45575 requested a full tablet report, sending...
I20260924 15:18:26.156651 4965 ts_manager.cc:194] Registered new tserver with Master: 7a1f636c2fb24437946b39b5d741b6af (127.31.12.66:46155)
I20260924 15:18:26.157037 4965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:50501
WARNING: no policy specified for kudu/127.31.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
I20260924 15:18:26.176147 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:45575
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:43375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:26.254915 5302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:26.255101 5302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:26.255123 5302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:26.256275 5302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:26.256320 5302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:26.256346 5302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:18:26.257825 5302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:43375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:45575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:26.258049 5302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:26.258306 5302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:26.263070 5308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:26.263156 5310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:26.263088 5302 server_base.cc:1034] running on GCE node
W20260924 15:18:26.263237 5307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:26.263485 5302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:26.263725 5302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:26.264915 5302 hybrid_clock.cc:648] HybridClock initialized: now 1790263106264880 us; error 32 us; skew 500 ppm
Sep 24 15:18:26 dist-test-slave-zwbb krb5kdc[4920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263106, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:26.267474 5302 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:18:26.267920 5302 webserver.cc:460] Webserver started at http://127.31.12.67:39033/ using document root <none> and password file <none>
I20260924 15:18:26.268160 5302 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:26.268227 5302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:26.268374 5302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:26.269356 5302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "625cba0ff97942e6826887fedb5cee16"
format_stamp: "Formatted at 2026-09-24 15:18:26 on dist-test-slave-zwbb"
server_key: "8cbd5b4db484fdd8e9227eaa7d48f694"
server_key_iv: "3d64e24451e0ea4fb3fa3608a398a648"
server_key_version: "encryptionkey@0"
I20260924 15:18:26.269788 5302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "625cba0ff97942e6826887fedb5cee16"
format_stamp: "Formatted at 2026-09-24 15:18:26 on dist-test-slave-zwbb"
server_key: "8cbd5b4db484fdd8e9227eaa7d48f694"
server_key_iv: "3d64e24451e0ea4fb3fa3608a398a648"
server_key_version: "encryptionkey@0"
I20260924 15:18:26.271190 5302 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:26.271898 5317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:26.272079 5302 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:26.272176 5302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "625cba0ff97942e6826887fedb5cee16"
format_stamp: "Formatted at 2026-09-24 15:18:26 on dist-test-slave-zwbb"
server_key: "8cbd5b4db484fdd8e9227eaa7d48f694"
server_key_iv: "3d64e24451e0ea4fb3fa3608a398a648"
server_key_version: "encryptionkey@0"
I20260924 15:18:26.272260 5302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:26.295792 5302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:26.298000 5302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:26.298161 5302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:26.298373 5302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:26.298687 5302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:26.298740 5302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:26.298782 5302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:26.298810 5302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:26.304574 5302 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:45621
I20260924 15:18:26.304620 5432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:45621 every 8 connection(s)
I20260924 15:18:26.304926 5302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:18:26 dist-test-slave-zwbb krb5kdc[4920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263106, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:18:26.310638 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 5302
I20260924 15:18:26.310719 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:18:26.310873 31793 external_mini_cluster.cc:1428] Setting key a69771679eaed7f2c30854805762dcbe
I20260924 15:18:26.313565 5006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:26.305651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:35605 (local address 127.31.12.126:45575)
0924 15:18:26.305727 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:26.305729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:26.305890 (+ 161us) server_negotiation.cc:407] Connection header received
0924 15:18:26.306770 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:26.306771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:26.306781 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:26.306834 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:26.307382 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:26.307867 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:26.308791 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:26.308967 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:26.309015 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:26.311101 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:26.311103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:26.311103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:26.311111 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:26.312467 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:26.312798 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:26.312799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:26.312799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:26.312847 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:26.313012 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:26.313012 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:26.313012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:26.313105 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:26.313150 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:26.313374 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 15:18:26.313431 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:18:26.313750 5435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:26.305622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:45575 (local address 127.31.12.67:35605)
0924 15:18:26.305777 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:26.305811 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:26.306683 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:26.306884 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:26.306887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:26.307134 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:26.307312 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:26.307316 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:26.307945 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:26.307945 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:26.308721 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:26.308723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:26.309036 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:26.309036 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:26.309129 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:26.309591 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:26.309599 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:26.311002 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:26.312535 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:26.312540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:26.312542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:26.312746 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:26.312888 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:26.312888 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:26.312888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:26.312973 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:26.313158 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:26.313162 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:26.313265 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 15:18:26.313343 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 15:18:26.313516 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":96,"threads_started":1}
I20260924 15:18:26.314226 5433 heartbeater.cc:347] Connected to a master server at 127.31.12.126:45575
I20260924 15:18:26.314318 5433 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:26.314497 5433 heartbeater.cc:511] Master 127.31.12.126:45575 requested a full tablet report, sending...
I20260924 15:18:26.314881 4965 ts_manager.cc:194] Registered new tserver with Master: 625cba0ff97942e6826887fedb5cee16 (127.31.12.67:45621)
I20260924 15:18:26.315251 4965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:35605
I20260924 15:18:26.322615 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/test-admin.keytab.
Sep 24 15:18:26 dist-test-slave-zwbb krb5kdc[4920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:26.338765 31793 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 15:18:26.340433 5440 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:18:26.339621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36502 (local address 127.31.12.65:37197)
0924 15:18:26.339789 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:26.339793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:26.339803 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:18:26.339826 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:26.339828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:26.339841 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:26.339919 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:26.340339 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36502: BlockingRecv error: recv got EOF from 127.0.0.1:36502 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
W20260924 15:18:26.340476 31793 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/krb5cc' of type FILE: Runtime error: Bad format in credentials cache (filename: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/krb5cc)
Sep 24 15:18:26 dist-test-slave-zwbb krb5kdc[4920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:26.341435 31793 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 15:18:26 dist-test-slave-zwbb krb5kdc[4920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:26.342362 31793 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 15:18:26 dist-test-slave-zwbb krb5kdc[4920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.65@KRBTEST.COM
I20260924 15:18:26.349432 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:26.342839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36518 (local address 127.31.12.65:37197)
0924 15:18:26.342893 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:26.342895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:26.342940 (+ 45us) server_negotiation.cc:407] Connection header received
0924 15:18:26.342980 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:26.342981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:26.342990 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:26.343035 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:26.343390 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:26.344077 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:26.344728 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:26.345049 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:26.345101 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:26.347066 (+ 1965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:26.347073 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:26.347074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:26.347080 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:26.348406 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:26.348713 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:26.348716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:26.348716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:26.348763 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:26.348917 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:26.348918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:26.348918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:26.349027 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:26.349115 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:26.349196 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 15:18:26.349251 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260924 15:18:26.351205 5440 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:18:26.350711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36526 (local address 127.31.12.65:37197)
0924 15:18:26.350748 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:26.350750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:26.350757 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:18:26.350797 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:26.350797 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:26.350805 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:26.350844 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:26.351177 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36526: BlockingRecv error: recv got EOF from 127.0.0.1:36526 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 15:18:26.351284 31793 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/krb5cc)
Sep 24 15:18:26 dist-test-slave-zwbb krb5kdc[4920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:26.352224 31793 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 15:18:26 dist-test-slave-zwbb krb5kdc[4920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:26.353060 31793 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 15:18:26 dist-test-slave-zwbb krb5kdc[4920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.65@KRBTEST.COM
I20260924 15:18:26.359683 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:26.353550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36532 (local address 127.31.12.65:37197)
0924 15:18:26.353610 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:26.353612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:26.353685 (+ 73us) server_negotiation.cc:407] Connection header received
0924 15:18:26.353754 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:26.353755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:26.353763 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:26.353830 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:26.354174 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:26.354642 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:26.355334 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:26.355535 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:26.355613 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:26.357558 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:26.357559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:26.357559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:26.357565 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:26.358788 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:26.359107 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:26.359108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:26.359108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:26.359155 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:26.359312 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:26.359313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:26.359313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:26.359404 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:26.359471 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:26.359579 (+ 108us) server_negotiation.cc:299] Negotiation successful
0924 15:18:26.359633 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260924 15:18:26.361287 5440 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:18:26.360829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36538 (local address 127.31.12.65:37197)
0924 15:18:26.360882 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:26.360884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:26.360911 (+ 27us) server_negotiation.cc:407] Connection header received
0924 15:18:26.360972 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:26.360972 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:26.360979 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:26.361025 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:26.361221 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36538: BlockingRecv error: recv got EOF from 127.0.0.1:36538 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260924 15:18:26.361407 31793 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790263080303576-31793-0/krb5kdc/krb5cc)
Sep 24 15:18:26 dist-test-slave-zwbb krb5kdc[4920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:26.362337 31793 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 15:18:26 dist-test-slave-zwbb krb5kdc[4920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:26.363164 31793 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 15:18:26 dist-test-slave-zwbb krb5kdc[4920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.65@KRBTEST.COM
I20260924 15:18:26.369514 5440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:26.363632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36554 (local address 127.31.12.65:37197)
0924 15:18:26.363696 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:26.363698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:26.363720 (+ 22us) server_negotiation.cc:407] Connection header received
0924 15:18:26.363776 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:26.363777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:26.363786 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:26.363852 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:26.364200 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:26.364634 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:26.365302 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:26.365570 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:26.365655 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:26.367474 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:26.367476 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:26.367476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:26.367481 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:26.368593 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:26.368878 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:26.368878 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:26.368879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:26.368926 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:26.369104 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:26.369105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:26.369105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:26.369179 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:26.369240 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:26.369324 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 15:18:26.369375 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 15:18:26.370128 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 5028
I20260924 15:18:26.375921 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 5164
I20260924 15:18:26.380440 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 5302
I20260924 15:18:26.384567 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 4933
2026-09-24T15:18:26Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1893 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:18:26 dist-test-slave-zwbb krb5kdc[5453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:18:26 dist-test-slave-zwbb krb5kdc[5453](info): set up 2 sockets
Sep 24 15:18:26 dist-test-slave-zwbb krb5kdc[5453](info): commencing operation
krb5kdc: starting...
W20260924 15:18:27.419258 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:18:27 dist-test-slave-zwbb krb5kdc[5453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:18:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:18:27Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.126.keytab.
Entry for principal oryx/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.126.keytab.
I20260924 15:18:27.532136 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:37657
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.126.keytab
--principal=oryx/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:37657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:27.611967 5466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:27.612214 5466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:27.612254 5466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:27.613415 5466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:18:27.613511 5466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:27.613544 5466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:27.613577 5466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:18:27.613601 5466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:18:27.615075 5466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:37657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.126.keytab
--principal=oryx/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:37657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:27.615351 5466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:27.615590 5466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:27.620473 5472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:27.620512 5474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:27.620620 5466 server_base.cc:1034] running on GCE node
W20260924 15:18:27.620471 5471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:27.621021 5466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:27.621361 5466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:27.622551 5466 hybrid_clock.cc:648] HybridClock initialized: now 1790263107622537 us; error 42 us; skew 500 ppm
Sep 24 15:18:27 dist-test-slave-zwbb krb5kdc[5453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263107, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:27.624961 5466 init.cc:377] Logged in from keytab as oryx/127.31.12.126@KRBTEST.COM (short username oryx)
I20260924 15:18:27.625381 5466 webserver.cc:460] Webserver started at http://127.31.12.126:40357/ using document root <none> and password file <none>
I20260924 15:18:27.625649 5466 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:27.625722 5466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:27.625877 5466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:27.626966 5466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "4b6f5737f73c4374a92ee2c8a69bac93"
format_stamp: "Formatted at 2026-09-24 15:18:27 on dist-test-slave-zwbb"
server_key: "cf52d0e242a344c968813d9601f6627c"
server_key_iv: "17509c63a387c0dc0f55e6949943fdf1"
server_key_version: "encryptionkey@0"
I20260924 15:18:27.627327 5466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "4b6f5737f73c4374a92ee2c8a69bac93"
format_stamp: "Formatted at 2026-09-24 15:18:27 on dist-test-slave-zwbb"
server_key: "cf52d0e242a344c968813d9601f6627c"
server_key_iv: "17509c63a387c0dc0f55e6949943fdf1"
server_key_version: "encryptionkey@0"
I20260924 15:18:27.628849 5466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:18:27.629647 5482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:27.629825 5466 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:27.629952 5466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "4b6f5737f73c4374a92ee2c8a69bac93"
format_stamp: "Formatted at 2026-09-24 15:18:27 on dist-test-slave-zwbb"
server_key: "cf52d0e242a344c968813d9601f6627c"
server_key_iv: "17509c63a387c0dc0f55e6949943fdf1"
server_key_version: "encryptionkey@0"
I20260924 15:18:27.630033 5466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:27.643433 5466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:27.645210 5466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:27.645372 5466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:27.649278 5466 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:37657
I20260924 15:18:27.649312 5534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:37657 every 8 connection(s)
I20260924 15:18:27.649740 5466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:18:27.650707 5535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:27.653607 5535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93: Bootstrap starting.
I20260924 15:18:27.654206 5535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:27.654515 5535 log.cc:824] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:27.655228 5535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93: No bootstrap required, opened a new log
I20260924 15:18:27.656541 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 5466
I20260924 15:18:27.656626 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:18:27.656822 31793 external_mini_cluster.cc:1428] Setting key e578fac868896ee342ab17bc2bdc4856
I20260924 15:18:27.656920 5535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } }
I20260924 15:18:27.657125 5535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:27.657173 5535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b6f5737f73c4374a92ee2c8a69bac93, State: Initialized, Role: FOLLOWER
I20260924 15:18:27.657305 5535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } }
I20260924 15:18:27.657399 5535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:18:27.657455 5535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:18:27.657508 5535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:27.658016 5535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } }
I20260924 15:18:27.658146 5535 leader_election.cc:302] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b6f5737f73c4374a92ee2c8a69bac93; no voters:
I20260924 15:18:27.658310 5535 leader_election.cc:288] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:18:27.658331 5540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:27.658434 5540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [term 1 LEADER]: Becoming Leader. State: Replica: 4b6f5737f73c4374a92ee2c8a69bac93, State: Running, Role: LEADER
I20260924 15:18:27.658511 5540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } }
I20260924 15:18:27.658571 5535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:18:27.658839 5540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b6f5737f73c4374a92ee2c8a69bac93. Latest consensus state: current_term: 1 leader_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } }
I20260924 15:18:27.658922 5540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:27.658849 5542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } }
I20260924 15:18:27.659113 5542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:27.659128 5545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:18:27.660125 5545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:18:27.662453 5545 catalog_manager.cc:1347] Generated new cluster ID: 789d17c9ecf04957b00f86660a385e57
I20260924 15:18:27.662509 5545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:18:27 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM
I20260924 15:18:27.671026 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:27.657421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54434 (local address 127.31.12.126:37657)
0924 15:18:27.657655 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:27.657662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:27.657685 (+ 23us) server_negotiation.cc:407] Connection header received
0924 15:18:27.658004 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:27.658010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:27.658052 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:27.658191 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:27.658941 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:27.660338 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:27.661146 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:27.661773 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:27.661889 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:27.667332 (+ 5443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:27.667336 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:27.667337 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:27.667355 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:27.669428 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:27.669851 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:27.669857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:27.669857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:27.669959 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:27.670167 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:27.670168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:27.670168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:27.670414 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:27.670605 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:27.670672 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 15:18:27.670809 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":92,"threads_started":1}
I20260924 15:18:27.679661 5545 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:18:27.680187 5545 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:18:27.686718 5545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93: Generated new TSK 0
I20260924 15:18:27.686887 5545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.65.keytab.
Entry for principal oryx/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.65.keytab.
I20260924 15:18:27.719954 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:37657
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.65.keytab
--principal=oryx/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:27.816820 5561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:27.817019 5561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:27.817042 5561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:27.818223 5561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:27.818265 5561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:27.818315 5561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:18:27.819804 5561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.65.keytab
--principal=oryx/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:37657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:27.820061 5561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:27.820302 5561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:18:27.825385 5561 server_base.cc:1034] running on GCE node
W20260924 15:18:27.825371 5566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:27.825352 5569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:27.825342 5567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:27.825820 5561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:27.826079 5561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:27.827265 5561 hybrid_clock.cc:648] HybridClock initialized: now 1790263107827224 us; error 57 us; skew 500 ppm
Sep 24 15:18:27 dist-test-slave-zwbb krb5kdc[5453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263107, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:27.829641 5561 init.cc:377] Logged in from keytab as oryx/127.31.12.65@KRBTEST.COM (short username oryx)
I20260924 15:18:27.830066 5561 webserver.cc:460] Webserver started at http://127.31.12.65:32917/ using document root <none> and password file <none>
I20260924 15:18:27.830299 5561 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:27.830366 5561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:27.830524 5561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:27.831530 5561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "27c5c2f9a521468c87e1cbbdcd730d7a"
format_stamp: "Formatted at 2026-09-24 15:18:27 on dist-test-slave-zwbb"
server_key: "9968b7ea2bd37807ae9e50dcc6e23308"
server_key_iv: "9328a74802ed583205ff65032e8245a9"
server_key_version: "encryptionkey@0"
I20260924 15:18:27.831909 5561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "27c5c2f9a521468c87e1cbbdcd730d7a"
format_stamp: "Formatted at 2026-09-24 15:18:27 on dist-test-slave-zwbb"
server_key: "9968b7ea2bd37807ae9e50dcc6e23308"
server_key_iv: "9328a74802ed583205ff65032e8245a9"
server_key_version: "encryptionkey@0"
I20260924 15:18:27.833261 5561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:18:27.834046 5577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:27.834213 5561 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:27.834311 5561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "27c5c2f9a521468c87e1cbbdcd730d7a"
format_stamp: "Formatted at 2026-09-24 15:18:27 on dist-test-slave-zwbb"
server_key: "9968b7ea2bd37807ae9e50dcc6e23308"
server_key_iv: "9328a74802ed583205ff65032e8245a9"
server_key_version: "encryptionkey@0"
I20260924 15:18:27.834404 5561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:27.845326 5561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:27.847199 5561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:27.847354 5561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:27.847573 5561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:27.847859 5561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:27.847919 5561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:27.847959 5561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:27.847986 5561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:27.853668 5561 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:43277
I20260924 15:18:27.853710 5690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:43277 every 8 connection(s)
I20260924 15:18:27.854010 5561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:18:27.855121 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 5561
I20260924 15:18:27.855228 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:18:27.855430 31793 external_mini_cluster.cc:1428] Setting key b3429dc001f9522d84b47af6ecc81922
Sep 24 15:18:27 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263107, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.31.12.66@KRBTEST.COM; defaulting to no policy
I20260924 15:18:27.864177 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:27.854694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:35407 (local address 127.31.12.126:37657)
0924 15:18:27.854739 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:27.854741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:27.854953 (+ 212us) server_negotiation.cc:407] Connection header received
0924 15:18:27.855638 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:27.855640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:27.855656 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:27.855723 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:27.856989 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:27.857585 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:27.858662 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:27.858882 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:27.858963 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:27.861208 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:27.861209 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:27.861210 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:27.861219 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:27.862843 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:27.863226 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:27.863228 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:27.863228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:27.863286 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:27.863529 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:27.863530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:27.863531 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:27.863671 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:27.863743 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:27.863982 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 15:18:27.864076 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:18:27.864262 5693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:27.854654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:35407)
0924 15:18:27.854804 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:27.854844 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:27.855538 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:27.855794 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:27.855798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:27.856575 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:27.856868 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:27.856888 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:27.857682 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:27.857683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:27.858574 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:27.858576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:27.858968 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:27.858970 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:27.859100 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:27.859716 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:27.859724 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:27.861120 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:27.862923 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:27.862929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:27.862931 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:27.863138 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:27.863356 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:27.863356 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:27.863357 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:27.863461 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:27.863737 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:27.863742 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:27.863845 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 15:18:27.863929 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 15:18:27.864081 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260924 15:18:27.864758 5691 heartbeater.cc:347] Connected to a master server at 127.31.12.126:37657
I20260924 15:18:27.864842 5691 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:27.865062 5691 heartbeater.cc:511] Master 127.31.12.126:37657 requested a full tablet report, sending...
I20260924 15:18:27.865636 5499 ts_manager.cc:194] Registered new tserver with Master: 27c5c2f9a521468c87e1cbbdcd730d7a (127.31.12.65:43277)
I20260924 15:18:27.866243 5499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:35407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.66.keytab.
Entry for principal oryx/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.66.keytab.
I20260924 15:18:27.880927 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:37657
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.66.keytab
--principal=oryx/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:27.962357 5696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:27.962550 5696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:27.962574 5696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:27.963668 5696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:27.963706 5696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:27.963732 5696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:18:27.965186 5696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.66.keytab
--principal=oryx/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:37657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:27.965369 5696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:27.965548 5696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:27.970486 5701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:27.970579 5702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:27.970487 5704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:27.970602 5696 server_base.cc:1034] running on GCE node
I20260924 15:18:27.970857 5696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:27.971048 5696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:27.972194 5696 hybrid_clock.cc:648] HybridClock initialized: now 1790263107972183 us; error 26 us; skew 500 ppm
Sep 24 15:18:27 dist-test-slave-zwbb krb5kdc[5453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263107, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:27.974736 5696 init.cc:377] Logged in from keytab as oryx/127.31.12.66@KRBTEST.COM (short username oryx)
I20260924 15:18:27.975165 5696 webserver.cc:460] Webserver started at http://127.31.12.66:40515/ using document root <none> and password file <none>
I20260924 15:18:27.975391 5696 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:27.975459 5696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:27.975611 5696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:27.976608 5696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "eb0abe6f3cda4468b8e17438d5d184bd"
format_stamp: "Formatted at 2026-09-24 15:18:27 on dist-test-slave-zwbb"
server_key: "752fbc5387dbaa59a8e85a2802ab9e3b"
server_key_iv: "9561bf3329b8a324d41a02724c620bdb"
server_key_version: "encryptionkey@0"
I20260924 15:18:27.976962 5696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "eb0abe6f3cda4468b8e17438d5d184bd"
format_stamp: "Formatted at 2026-09-24 15:18:27 on dist-test-slave-zwbb"
server_key: "752fbc5387dbaa59a8e85a2802ab9e3b"
server_key_iv: "9561bf3329b8a324d41a02724c620bdb"
server_key_version: "encryptionkey@0"
I20260924 15:18:27.978339 5696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:18:27.979063 5711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:27.979228 5696 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:27.979326 5696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "eb0abe6f3cda4468b8e17438d5d184bd"
format_stamp: "Formatted at 2026-09-24 15:18:27 on dist-test-slave-zwbb"
server_key: "752fbc5387dbaa59a8e85a2802ab9e3b"
server_key_iv: "9561bf3329b8a324d41a02724c620bdb"
server_key_version: "encryptionkey@0"
I20260924 15:18:27.979408 5696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:28.011691 5696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:28.014094 5696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:28.014256 5696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:28.014492 5696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:28.014788 5696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:28.014840 5696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.014881 5696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:28.014909 5696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.020721 5696 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:37031
I20260924 15:18:28.020762 5826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:37031 every 8 connection(s)
I20260924 15:18:28.021071 5696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:18:28.025885 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 5696
I20260924 15:18:28.026005 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:18:28.026198 31793 external_mini_cluster.cc:1428] Setting key 5f059679adf1807382c270022881b411
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263107, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM
I20260924 15:18:28.030839 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.021788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:53945 (local address 127.31.12.126:37657)
0924 15:18:28.021846 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.021847 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.022055 (+ 208us) server_negotiation.cc:407] Connection header received
0924 15:18:28.022510 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.022510 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.022520 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.022596 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.023457 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.023952 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.024862 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.025043 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.025093 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.027610 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.027611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.027611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.027617 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.029305 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.029787 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.029788 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.029788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.029859 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.030074 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.030075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.030075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.030222 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.030291 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.030597 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.030702 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 15:18:28.030840 5829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.021746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:53945)
0924 15:18:28.021889 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.021941 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.022430 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.022649 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.022652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.023199 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.023383 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.023387 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.024028 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.024029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.024782 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.024784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.025110 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.025111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.025206 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.025737 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.025746 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.027510 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.029483 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.029489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.029491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.029712 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.029929 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.029929 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.029929 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.030011 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.030305 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.030311 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.030456 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.030541 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.030652 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260924 15:18:28.031803 5827 heartbeater.cc:347] Connected to a master server at 127.31.12.126:37657
I20260924 15:18:28.031883 5827 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:28.032105 5827 heartbeater.cc:511] Master 127.31.12.126:37657 requested a full tablet report, sending...
I20260924 15:18:28.032523 5499 ts_manager.cc:194] Registered new tserver with Master: eb0abe6f3cda4468b8e17438d5d184bd (127.31.12.66:37031)
I20260924 15:18:28.032893 5499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:53945
WARNING: no policy specified for oryx/127.31.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.67.keytab.
Entry for principal oryx/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.67.keytab.
I20260924 15:18:28.050060 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:37657
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.67.keytab
--principal=oryx/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:28.129215 5832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:28.129423 5832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:28.129470 5832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:28.130642 5832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:28.130708 5832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:28.130749 5832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:18:28.132272 5832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.67.keytab
--principal=oryx/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:37657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:28.132495 5832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:28.132675 5832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:28.137769 5838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:28.137790 5837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:28.137908 5832 server_base.cc:1034] running on GCE node
W20260924 15:18:28.137761 5840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:28.138209 5832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:28.138438 5832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:28.139599 5832 hybrid_clock.cc:648] HybridClock initialized: now 1790263108139583 us; error 46 us; skew 500 ppm
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263108, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:28.142036 5832 init.cc:377] Logged in from keytab as oryx/127.31.12.67@KRBTEST.COM (short username oryx)
I20260924 15:18:28.142484 5832 webserver.cc:460] Webserver started at http://127.31.12.67:45911/ using document root <none> and password file <none>
I20260924 15:18:28.142726 5832 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:28.142794 5832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:28.142949 5832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:28.143923 5832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "a2dfc048810b439d880bf058dfda0a06"
format_stamp: "Formatted at 2026-09-24 15:18:28 on dist-test-slave-zwbb"
server_key: "b6853d24b905e98d3bfe4ddbc55f724d"
server_key_iv: "d912a5cf5cc7a208955a9f4912f13fca"
server_key_version: "encryptionkey@0"
I20260924 15:18:28.144274 5832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "a2dfc048810b439d880bf058dfda0a06"
format_stamp: "Formatted at 2026-09-24 15:18:28 on dist-test-slave-zwbb"
server_key: "b6853d24b905e98d3bfe4ddbc55f724d"
server_key_iv: "d912a5cf5cc7a208955a9f4912f13fca"
server_key_version: "encryptionkey@0"
I20260924 15:18:28.145658 5832 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:18:28.146379 5847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.146559 5832 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:28.146639 5832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "a2dfc048810b439d880bf058dfda0a06"
format_stamp: "Formatted at 2026-09-24 15:18:28 on dist-test-slave-zwbb"
server_key: "b6853d24b905e98d3bfe4ddbc55f724d"
server_key_iv: "d912a5cf5cc7a208955a9f4912f13fca"
server_key_version: "encryptionkey@0"
I20260924 15:18:28.146690 5832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:28.160534 5832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:28.162709 5832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:28.162861 5832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:28.163072 5832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:28.163360 5832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:28.163393 5832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.163412 5832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:28.163455 5832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.169322 5832 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:32865
I20260924 15:18:28.169368 5961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:32865 every 8 connection(s)
I20260924 15:18:28.169701 5832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:18:28.174579 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 5832
I20260924 15:18:28.174675 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:18:28.174846 31793 external_mini_cluster.cc:1428] Setting key 9caf170e932fc3a711d467f1ef755867
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263108, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM
I20260924 15:18:28.178539 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.170366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:54821 (local address 127.31.12.126:37657)
0924 15:18:28.170410 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.170412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.170583 (+ 171us) server_negotiation.cc:407] Connection header received
0924 15:18:28.171253 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.171255 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.171265 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.171315 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.172221 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.172696 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.173734 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.173919 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.173972 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.176273 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.176274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.176274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.176280 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.177404 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.177817 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.177818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.177818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.177869 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.178067 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.178067 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.178068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.178166 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.178214 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.178412 (+ 198us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.178476 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:18:28.178743 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.170326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:54821)
0924 15:18:28.170457 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.170503 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.171134 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.171378 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.171381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.171941 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.172139 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.172144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.172781 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.172781 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.173657 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.173660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.173991 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.173992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.174086 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.174523 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.174535 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.176197 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.177540 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.177551 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.177552 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.177756 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.177926 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.177927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.177927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.178016 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.178218 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.178222 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.178324 (+ 102us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.178400 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.178508 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260924 15:18:28.179215 5962 heartbeater.cc:347] Connected to a master server at 127.31.12.126:37657
I20260924 15:18:28.179303 5962 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:28.179428 5962 heartbeater.cc:511] Master 127.31.12.126:37657 requested a full tablet report, sending...
I20260924 15:18:28.179788 5499 ts_manager.cc:194] Registered new tserver with Master: a2dfc048810b439d880bf058dfda0a06 (127.31.12.67:32865)
I20260924 15:18:28.180147 5499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:54821
I20260924 15:18:28.186630 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.12.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.12.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.12.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.125.keytab.
Entry for principal oryx/127.31.12.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.125.keytab.
I20260924 15:18:28.210023 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.125:35423
--webserver_interface=127.31.12.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.125.keytab
--principal=oryx/127.31.12.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:37657,127.31.12.125:35423
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:28.289393 5967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:28.289656 5967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:28.289685 5967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:28.290769 5967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:18:28.290802 5967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:28.290817 5967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:28.290829 5967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:18:28.290841 5967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:18:28.292227 5967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:37657,127.31.12.125:35423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.125.keytab
--principal=oryx/127.31.12.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.125:35423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.12.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
W20260924 15:18:28.292325 5967 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.31.12.126:37657,127.31.12.125:35423'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 15:18:28.292433 5967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:28.292598 5967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:28.297204 5972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:28.297194 5975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:28.297405 5967 server_base.cc:1034] running on GCE node
W20260924 15:18:28.297570 5973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:28.297844 5967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:28.298097 5967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:28.299266 5967 hybrid_clock.cc:648] HybridClock initialized: now 1790263108299248 us; error 36 us; skew 500 ppm
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263108, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:28.301800 5967 init.cc:377] Logged in from keytab as oryx/127.31.12.125@KRBTEST.COM (short username oryx)
I20260924 15:18:28.302227 5967 webserver.cc:460] Webserver started at http://127.31.12.125:39461/ using document root <none> and password file <none>
I20260924 15:18:28.302465 5967 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:28.302534 5967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:28.302688 5967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:28.303716 5967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/data/instance:
uuid: "efed9eeea550446e838d24f9dc145216"
format_stamp: "Formatted at 2026-09-24 15:18:28 on dist-test-slave-zwbb"
server_key: "718500ef67e5792b1012e73ba23a5ba6"
server_key_iv: "a7289f3fa9a91d877f1de49b246755b1"
server_key_version: "encryptionkey@0"
I20260924 15:18:28.304082 5967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/wal/instance:
uuid: "efed9eeea550446e838d24f9dc145216"
format_stamp: "Formatted at 2026-09-24 15:18:28 on dist-test-slave-zwbb"
server_key: "718500ef67e5792b1012e73ba23a5ba6"
server_key_iv: "a7289f3fa9a91d877f1de49b246755b1"
server_key_version: "encryptionkey@0"
I20260924 15:18:28.305589 5967 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:18:28.306365 5982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.306538 5967 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:28.306645 5967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/wal
uuid: "efed9eeea550446e838d24f9dc145216"
format_stamp: "Formatted at 2026-09-24 15:18:28 on dist-test-slave-zwbb"
server_key: "718500ef67e5792b1012e73ba23a5ba6"
server_key_iv: "a7289f3fa9a91d877f1de49b246755b1"
server_key_version: "encryptionkey@0"
I20260924 15:18:28.306726 5967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:28.321768 5967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:28.324080 5967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:28.324247 5967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:28.328271 5967 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.125:35423
I20260924 15:18:28.328307 6034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.125:35423 every 8 connection(s)
I20260924 15:18:28.328655 5967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/data/info.pb
I20260924 15:18:28.329718 6035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:28.331218 6035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:18:28.334379 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 5967
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263108, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.125@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.12.125@KRBTEST.COM
I20260924 15:18:28.341194 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.331894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54446 (local address 127.31.12.126:37657)
0924 15:18:28.331977 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.331982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.332200 (+ 218us) server_negotiation.cc:407] Connection header received
0924 15:18:28.332659 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.332660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.332670 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.332715 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.333354 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.334074 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.335473 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.335661 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.335708 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.338109 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.338110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.338111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.338119 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.339732 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.340196 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.340198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.340198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.340262 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.340505 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.340506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.340507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.340676 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.340728 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.341066 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.341131 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 15:18:28.341534 6037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.331873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.0.0.1:54446)
0924 15:18:28.332086 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.332122 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.332562 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.332788 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.332792 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.333093 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.333285 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.333290 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.334175 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.334176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.335384 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.335387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.335743 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.335744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.335900 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.336408 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.336417 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.338020 (+ 1603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.339817 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.339823 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.339825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.340029 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.340334 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.340334 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.340335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.340433 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.340745 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.340750 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.340885 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.341036 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.341240 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":125,"threads_started":1}
I20260924 15:18:28.342314 6035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:18:28.343562 6039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.334887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43028 (local address 127.31.12.125:35423)
0924 15:18:28.335042 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.335046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.335065 (+ 19us) server_negotiation.cc:407] Connection header received
0924 15:18:28.335111 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.335113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.335128 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.335183 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.335865 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.336524 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.337455 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.337832 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.337922 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.340574 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.340580 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.340580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.340585 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.342335 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.342745 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.342748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.342748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.342806 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.343031 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.343032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.343032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.343209 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.343343 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.343440 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.343497 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":104,"threads_started":1}
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263108, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.125@KRBTEST.COM for oryx/127.31.12.125@KRBTEST.COM
I20260924 15:18:28.348866 6037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.342573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:35423 (local address 127.0.0.1:43034)
0924 15:18:28.342616 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.342625 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.342658 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.342936 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.342937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.343078 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.343130 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.343132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.344006 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.344007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.344524 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.344525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.344869 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.344870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.344922 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.345215 (+ 293us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.345218 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.346340 (+ 1122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.348109 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.348110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.348110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.348262 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.348425 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.348425 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.348426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.348483 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.348667 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.348668 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.348698 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.348746 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.348812 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 15:18:28.349079 6040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.342628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43034 (local address 127.31.12.125:35423)
0924 15:18:28.342792 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.342795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.342801 (+ 6us) server_negotiation.cc:407] Connection header received
0924 15:18:28.342817 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.342817 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.342825 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.342917 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.343200 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.343930 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.344598 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.344796 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.344849 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.346410 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.346411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.346411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.346417 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.348030 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.348323 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.348324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.348324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.348370 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.348528 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.348529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.348529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.348612 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.348663 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.348759 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.348855 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
I20260924 15:18:28.350407 6035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216: Bootstrap starting.
I20260924 15:18:28.350966 6035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:28.351182 6035 log.cc:824] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:28.351804 6035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216: No bootstrap required, opened a new log
I20260924 15:18:28.353142 6035 raft_consensus.cc:348] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } }
I20260924 15:18:28.353257 6035 raft_consensus.cc:374] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:28.353278 6035 raft_consensus.cc:729] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efed9eeea550446e838d24f9dc145216, State: Initialized, Role: FOLLOWER
I20260924 15:18:28.353394 6035 consensus_queue.cc:260] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } }
I20260924 15:18:28.353693 6035 sys_catalog.cc:564] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:18:28.353662 6042 sys_catalog.cc:455] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } } }
I20260924 15:18:28.353763 6042 sys_catalog.cc:458] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 15:18:28.355288 6053 catalog_manager.cc:1558] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:18:28.355348 6053 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:18:28.356879 5967 master_runner.cc:428] Detected that this master efed9eeea550446e838d24f9dc145216 is joining an existing cluster
I20260924 15:18:28.356928 5967 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.12.125:35423
I20260924 15:18:28.364920 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.357998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54448 (local address 127.31.12.126:37657)
0924 15:18:28.358073 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.358076 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.358211 (+ 135us) server_negotiation.cc:407] Connection header received
0924 15:18:28.358262 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.358263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.358276 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.358334 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.358770 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.359588 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.360639 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.360891 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.360968 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.361949 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.361950 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.361951 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.361958 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.363621 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.364092 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.364093 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.364094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.364152 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.364365 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.364366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.364366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.364492 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.364560 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.364739 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.364811 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 15:18:28.365033 6040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.358287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43044 (local address 127.31.12.125:35423)
0924 15:18:28.358320 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.358322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.358375 (+ 53us) server_negotiation.cc:407] Connection header received
0924 15:18:28.358490 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.358491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.358500 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.358526 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.358982 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.359710 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.360403 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.360646 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.360711 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.361938 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.361946 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.361946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.361952 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.363647 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.364142 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.364144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.364144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.364199 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.364420 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.364421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.364421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.364530 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.364570 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.364738 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.364906 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":16}
I20260924 15:18:28.364945 6059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.357955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.0.0.1:54448)
0924 15:18:28.358131 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.358142 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.358207 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.358381 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.358382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.358605 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.358686 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.358688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.359669 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.359670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.360551 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.360553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.360966 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.360967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.361030 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.361497 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.361502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.361839 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.363725 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.363726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.363726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.364001 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.364224 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.364225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.364226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.364285 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.364587 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.364588 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.364631 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.364711 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.364823 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":71,"threads_started":1}
I20260924 15:18:28.365643 6060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.358160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:35423 (local address 127.0.0.1:43044)
0924 15:18:28.358335 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.358343 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.358387 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.358588 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.358589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.358849 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.358901 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.358903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.359810 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.359810 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.360321 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.360322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.360733 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.360734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.360795 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.361352 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.361358 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.361839 (+ 481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.363733 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.363734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.363734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.364060 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.364276 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.364277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.364277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.364354 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.364611 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.364612 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.364644 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.364764 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.365551 (+ 787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":169,"thread_start_us":64,"threads_started":1}
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263107, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.126@KRBTEST.COM for oryx/127.31.12.125@KRBTEST.COM
I20260924 15:18:28.374153 6040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.367291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43060 (local address 127.31.12.125:35423)
0924 15:18:28.367326 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.367329 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.367476 (+ 147us) server_negotiation.cc:407] Connection header received
0924 15:18:28.367568 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.367569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.367580 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.367635 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.368040 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.368495 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.369263 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.369534 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.369624 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.371630 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.371631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.371631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.371637 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.372881 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.373306 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.373307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.373307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.373367 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.373614 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.373615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.373615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.373721 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.373791 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.374001 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.374069 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 15:18:28.374181 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.367242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:35423 (local address 127.0.0.1:43060)
0924 15:18:28.367368 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.367416 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.367501 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.367641 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.367644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.367906 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.367959 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.367961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.368583 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.368585 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.369175 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.369177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.369628 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.369629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.369710 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.370119 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.370123 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.371537 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.372978 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.372986 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.372989 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.373220 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.373463 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.373464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.373464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.373546 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.373798 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.373807 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.373856 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.373947 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.374081 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
I20260924 15:18:28.374900 5499 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.12.125:35423: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "efed9eeea550446e838d24f9dc145216"
member_type: NON_VOTER
last_known_addr {
host: "127.31.12.125"
port: 35423
}
attrs {
promote: true
}
}
dest_uuid: "4b6f5737f73c4374a92ee2c8a69bac93"
cas_config_opid_index: -1
I20260924 15:18:28.375227 5499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: NON_VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: true } }
I20260924 15:18:28.375691 6063 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER efed9eeea550446e838d24f9dc145216 (127.31.12.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: NON_VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: true } } }
I20260924 15:18:28.376022 6009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:28.376784 6009 raft_consensus.cc:1261] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 FOLLOWER]: Refusing update from remote peer 4b6f5737f73c4374a92ee2c8a69bac93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 15:18:28.377151 6062 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: NON_VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:28.377878 6063 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.12.125:35423
I20260924 15:18:28.377964 6063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: NON_VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: true } } }
I20260924 15:18:28.378021 6063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:28.378098 5967 master.cc:537] Master@127.31.12.125:35423 shutting down...
I20260924 15:18:28.379230 6009 raft_consensus.cc:2932] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER efed9eeea550446e838d24f9dc145216 (127.31.12.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: NON_VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: true } } }
I20260924 15:18:28.380097 6071 sys_catalog.cc:455] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: NON_VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: true } } }
I20260924 15:18:28.380414 6042 sys_catalog.cc:455] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b6f5737f73c4374a92ee2c8a69bac93. Latest consensus state: current_term: 1 leader_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: NON_VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: true } } }
I20260924 15:18:28.380491 6042 sys_catalog.cc:458] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [sys.catalog]: This master's current role is: LEARNER
I20260924 15:18:28.380419 6067 sys_catalog.cc:455] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: NON_VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: true } } }
I20260924 15:18:28.380306 6071 sys_catalog.cc:458] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [sys.catalog]: This master's current role is: LEARNER
I20260924 15:18:28.380548 6067 sys_catalog.cc:458] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [sys.catalog]: This master's current role is: LEARNER
I20260924 15:18:28.382823 5967 raft_consensus.cc:2229] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 LEARNER]: Raft consensus shutting down.
I20260924 15:18:28.382936 5967 raft_consensus.cc:2258] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 LEARNER]: Raft consensus is shut down!
I20260924 15:18:28.382978 5967 tablet_replica.cc:330] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216: stopping tablet replica
I20260924 15:18:28.395849 5967 master.cc:552] Master@127.31.12.125:35423 shutdown complete.
I20260924 15:18:28.395920 5967 master_runner.cc:305] Clearing existing system tablet
I20260924 15:18:28.396056 5967 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:28.396117 5967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:28.396718 5967 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:18:28.397226 6074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.397403 5967 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:28.397492 5967 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:18:28.397521 5967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/wal
uuid: "efed9eeea550446e838d24f9dc145216"
format_stamp: "Formatted at 2026-09-24 15:18:28 on dist-test-slave-zwbb"
server_key: "718500ef67e5792b1012e73ba23a5ba6"
server_key_iv: "a7289f3fa9a91d877f1de49b246755b1"
server_key_version: "encryptionkey@0"
I20260924 15:18:28.398264 5967 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:28.399838 5967 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 15:18:28.399919 5967 log.cc:1196] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 15:18:28.400159 5967 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216: Deleting consensus metadata
I20260924 15:18:28.400277 5967 master_runner.cc:315] Copying system tablet from 127.31.12.126:37657
I20260924 15:18:28.400882 5967 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216: tablet copy: Beginning tablet copy session from remote peer at address 127.31.12.126:37657
I20260924 15:18:28.407421 6080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.401177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.0.0.1:54460)
0924 15:18:28.401284 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.401295 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.401355 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.401633 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.401633 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.401780 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.401835 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.401837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.402796 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.402796 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.403333 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.403335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.403820 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.403820 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.403868 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.404350 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.404354 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.404635 (+ 281us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.406495 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.406496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.406496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.406665 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.406865 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.406866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.406866 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.406917 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.407183 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.407183 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.407218 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.407270 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.407346 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":70,"threads_started":1}
I20260924 15:18:28.407490 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.401246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54460 (local address 127.31.12.126:37657)
0924 15:18:28.401299 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.401301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.401347 (+ 46us) server_negotiation.cc:407] Connection header received
0924 15:18:28.401409 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.401410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.401421 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.401629 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.401912 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.402710 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.403414 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.403739 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.403816 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.404864 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.404866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.404866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.404874 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.406419 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.406737 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.406738 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.406739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.406802 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.406978 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.406979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.406979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.407120 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.407189 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.407309 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.407395 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 15:18:28.407806 5519 tablet_copy_service.cc:140] P 4b6f5737f73c4374a92ee2c8a69bac93: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer efed9eeea550446e838d24f9dc145216 ({username='oryx', principal='oryx/127.31.12.125@KRBTEST.COM'} at 127.0.0.1:54460)
I20260924 15:18:28.407907 5519 tablet_copy_service.cc:161] P 4b6f5737f73c4374a92ee2c8a69bac93: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer efed9eeea550446e838d24f9dc145216 at {username='oryx', principal='oryx/127.31.12.125@KRBTEST.COM'} at 127.0.0.1:54460: session id = efed9eeea550446e838d24f9dc145216-00000000000000000000000000000000
I20260924 15:18:28.408717 5519 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 15:18:28.409302 5967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 15:18:28.410218 5967 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216: tablet copy: Starting download of 0 data blocks...
I20260924 15:18:28.410378 5967 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216: tablet copy: Starting download of 1 WAL segments...
I20260924 15:18:28.411276 5967 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 15:18:28.412055 5519 tablet_copy_service.cc:342] P 4b6f5737f73c4374a92ee2c8a69bac93: Request end of tablet copy session efed9eeea550446e838d24f9dc145216-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.12.125@KRBTEST.COM'} at 127.0.0.1:54460
I20260924 15:18:28.412119 5519 tablet_copy_service.cc:433] P 4b6f5737f73c4374a92ee2c8a69bac93: ending tablet copy session efed9eeea550446e838d24f9dc145216-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer efed9eeea550446e838d24f9dc145216
I20260924 15:18:28.414574 5967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:28.415688 6081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:28.415812 6082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:28.415874 6084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:28.416095 5967 server_base.cc:1034] running on GCE node
I20260924 15:18:28.416183 5967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:28.416330 5967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:28.417505 5967 hybrid_clock.cc:648] HybridClock initialized: now 1790263108417491 us; error 27 us; skew 500 ppm
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263108, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:28.418777 5967 init.cc:377] Logged in from keytab as oryx/127.31.12.125@KRBTEST.COM (short username oryx)
I20260924 15:18:28.419023 5967 webserver.cc:460] Webserver started at http://127.31.12.125:44483/ using document root <none> and password file <none>
I20260924 15:18:28.419149 5967 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:28.419200 5967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:28.419791 5967 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.420370 6091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.420500 5967 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.420573 5967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/wal
uuid: "efed9eeea550446e838d24f9dc145216"
format_stamp: "Formatted at 2026-09-24 15:18:28 on dist-test-slave-zwbb"
server_key: "718500ef67e5792b1012e73ba23a5ba6"
server_key_iv: "a7289f3fa9a91d877f1de49b246755b1"
server_key_version: "encryptionkey@0"
I20260924 15:18:28.420632 5967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:28.439970 5967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:28.441762 5967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:28.444794 5967 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.125:35423
I20260924 15:18:28.444833 6143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.125:35423 every 8 connection(s)
I20260924 15:18:28.445214 5967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-1/data/info.pb
I20260924 15:18:28.446221 6145 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:18:28.446641 6145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216: Bootstrap starting.
I20260924 15:18:28.448624 6145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:18:28.448849 6145 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216: Bootstrap complete.
I20260924 15:18:28.449067 6145 raft_consensus.cc:348] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: NON_VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: true } }
I20260924 15:18:28.449218 6145 raft_consensus.cc:729] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: efed9eeea550446e838d24f9dc145216, State: Initialized, Role: LEARNER
I20260924 15:18:28.449281 6145 consensus_queue.cc:260] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: NON_VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: true } }
I20260924 15:18:28.449546 6145 sys_catalog.cc:564] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:18:28.449533 6148 sys_catalog.cc:455] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: NON_VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: true } } }
I20260924 15:18:28.449759 6148 sys_catalog.cc:458] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [sys.catalog]: This master's current role is: LEARNER
I20260924 15:18:28.451203 6159 catalog_manager.cc:1259] Loaded cluster ID: 789d17c9ecf04957b00f86660a385e57
I20260924 15:18:28.451257 6159 catalog_manager.cc:1552] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216: loading cluster ID for follower catalog manager: success
I20260924 15:18:28.452309 6159 catalog_manager.cc:1574] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216: acquiring CA information for follower catalog manager: success
I20260924 15:18:28.452641 6159 catalog_manager.cc:1602] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:18:28.454594 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.445249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43074 (local address 127.31.12.125:35423)
0924 15:18:28.445404 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.445406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.445415 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:18:28.445472 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.445473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.445485 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.445588 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.446233 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.447205 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.448163 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.448473 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.448548 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.450050 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.450052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.450052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.450059 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.453428 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.453879 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.453880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.453880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.453930 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.454134 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.454134 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.454135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.454226 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.454296 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.454456 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.454526 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for oryx/127.31.12.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.12.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.12.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.124.keytab.
Entry for principal oryx/127.31.12.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.124.keytab.
I20260924 15:18:28.478557 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.124:40647
--webserver_interface=127.31.12.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.124.keytab
--principal=oryx/127.31.12.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:37657,127.31.12.125:35423,127.31.12.124:40647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:18:28.559984 6162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:28.560238 6162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:28.560273 6162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:28.561378 6162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:18:28.561439 6162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:28.561484 6162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:28.561506 6162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:18:28.561532 6162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:18:28.563158 6162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:37657,127.31.12.125:35423,127.31.12.124:40647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.124.keytab
--principal=oryx/127.31.12.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.124:40647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.12.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:28.563416 6162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:28.563642 6162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:28.568511 6167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:28.568527 6170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:28.568560 6162 server_base.cc:1034] running on GCE node
W20260924 15:18:28.568527 6168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:28.568933 6162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:28.569214 6162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:28.570374 6162 hybrid_clock.cc:648] HybridClock initialized: now 1790263108570339 us; error 56 us; skew 500 ppm
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263108, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:28.572722 6162 init.cc:377] Logged in from keytab as oryx/127.31.12.124@KRBTEST.COM (short username oryx)
I20260924 15:18:28.573174 6162 webserver.cc:460] Webserver started at http://127.31.12.124:46443/ using document root <none> and password file <none>
I20260924 15:18:28.573352 6162 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:28.573390 6162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:28.573537 6162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:18:28.574504 6162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/data/instance:
uuid: "0cc1c8105976413c9be36d2d9f807985"
format_stamp: "Formatted at 2026-09-24 15:18:28 on dist-test-slave-zwbb"
server_key: "78e03bc8b146b74057445681df3fe29b"
server_key_iv: "6877ad06bb51b5fc7aae99dc0166e5a9"
server_key_version: "encryptionkey@0"
I20260924 15:18:28.574821 6162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/wal/instance:
uuid: "0cc1c8105976413c9be36d2d9f807985"
format_stamp: "Formatted at 2026-09-24 15:18:28 on dist-test-slave-zwbb"
server_key: "78e03bc8b146b74057445681df3fe29b"
server_key_iv: "6877ad06bb51b5fc7aae99dc0166e5a9"
server_key_version: "encryptionkey@0"
I20260924 15:18:28.576052 6162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:18:28.576674 6177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.576841 6162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:18:28.576920 6162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/wal
uuid: "0cc1c8105976413c9be36d2d9f807985"
format_stamp: "Formatted at 2026-09-24 15:18:28 on dist-test-slave-zwbb"
server_key: "78e03bc8b146b74057445681df3fe29b"
server_key_iv: "6877ad06bb51b5fc7aae99dc0166e5a9"
server_key_version: "encryptionkey@0"
I20260924 15:18:28.576973 6162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:28.585534 6162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:28.587150 6162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:28.587255 6162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:28.590888 6162 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.124:40647
I20260924 15:18:28.590931 6230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.124:40647 every 8 connection(s)
I20260924 15:18:28.591257 6162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/data/info.pb
I20260924 15:18:28.592222 6231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:28.593204 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 6162
I20260924 15:18:28.593928 6231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263108, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.124@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.12.124@KRBTEST.COM
I20260924 15:18:28.603683 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.595102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54462 (local address 127.31.12.126:37657)
0924 15:18:28.595168 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.595170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.595264 (+ 94us) server_negotiation.cc:407] Connection header received
0924 15:18:28.595333 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.595334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.595342 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.595389 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.595870 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.596610 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.597869 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.598166 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.598230 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.600566 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.600568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.600568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.600584 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.602174 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.602673 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.602674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.602675 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.602726 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.602982 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.602983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.602983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.603109 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.603180 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.603454 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.603555 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 15:18:28.603827 6234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.594892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.0.0.1:54462)
0924 15:18:28.595140 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.595186 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.595266 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.595441 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.595443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.595680 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.595773 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.595777 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.596729 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.596730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.597796 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.597799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.598270 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.598271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.598346 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.598846 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.598857 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.600466 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.602252 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.602259 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.602261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.602590 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.602790 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.602791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.602791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.602916 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.603185 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.603189 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.603302 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.603416 (+ 114us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.603585 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":64,"threads_started":1}
I20260924 15:18:28.604434 6231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:18:28.604528 6233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.593816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53480 (local address 127.31.12.124:40647)
0924 15:18:28.594006 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.594014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.594037 (+ 23us) server_negotiation.cc:407] Connection header received
0924 15:18:28.594457 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.594463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.594507 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.594661 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.595448 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.596659 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.597683 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.598077 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.598181 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.601341 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.601347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.601347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.601354 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.603052 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.603386 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.603388 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.603389 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.603443 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.603646 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.603647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.603648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.603790 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.604324 (+ 534us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.604370 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.604443 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":88,"threads_started":1}
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263108, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.124@KRBTEST.COM for oryx/127.31.12.125@KRBTEST.COM
I20260924 15:18:28.610653 6234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.604918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:35423 (local address 127.0.0.1:43082)
0924 15:18:28.605052 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.605062 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.605114 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.605339 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.605339 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.605499 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.605548 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.605550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.606210 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.606212 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.606716 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.606718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.607102 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.607103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.607167 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.607486 (+ 319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.607490 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.608587 (+ 1097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.609915 (+ 1328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.609916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.609916 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.610077 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.610228 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.610228 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.610228 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.610268 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.610455 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.610456 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.610486 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.610536 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.610598 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260924 15:18:28.610666 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.605005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43082 (local address 127.31.12.125:35423)
0924 15:18:28.605071 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.605073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.605128 (+ 55us) server_negotiation.cc:407] Connection header received
0924 15:18:28.605265 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.605267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.605278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.605353 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.605646 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.606134 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.606800 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.607018 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.607102 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.608662 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.608663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.608663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.608670 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.609849 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.610138 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.610139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.610139 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.610184 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.610313 (+ 129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.610313 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.610314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.610403 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.610461 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.610551 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.610608 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 15:18:28.611061 6231 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.124" port: 40647 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263108, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.124@KRBTEST.COM for oryx/127.31.12.124@KRBTEST.COM
I20260924 15:18:28.616478 6234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.611311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:40647 (local address 127.0.0.1:53488)
0924 15:18:28.611356 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.611366 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.611400 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.611501 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.611502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.611634 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.611655 (+ 21us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.611656 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.612278 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.612279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.612749 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.612750 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.613096 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.613096 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.613142 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.613523 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.613526 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.614506 (+ 980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.615742 (+ 1236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.615743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.615743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.615890 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.616064 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.616065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.616065 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.616112 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.616271 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.616271 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.616301 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.616349 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.616414 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 15:18:28.616529 6233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.611378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53488 (local address 127.31.12.124:40647)
0924 15:18:28.611415 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.611417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.611424 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:18:28.611450 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.611451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.611458 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.611500 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.611720 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.612192 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.612833 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.613029 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.613099 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.614574 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.614575 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.614575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.614580 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.615677 (+ 1097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.615965 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.615966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.615966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.616009 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.616148 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.616148 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.616148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.616224 (+ 76us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.616266 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.616362 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.616453 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":22}
I20260924 15:18:28.617892 6231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985: Bootstrap starting.
I20260924 15:18:28.618696 6231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:28.618971 6231 log.cc:824] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:28.619685 6231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985: No bootstrap required, opened a new log
I20260924 15:18:28.621153 6231 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 40647 } }
I20260924 15:18:28.621304 6231 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:28.621356 6231 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cc1c8105976413c9be36d2d9f807985, State: Initialized, Role: FOLLOWER
I20260924 15:18:28.621582 6231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 40647 } }
I20260924 15:18:28.621963 6231 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:18:28.621918 6236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 40647 } } }
I20260924 15:18:28.622090 6236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 15:18:28.623627 6247 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:18:28.623687 6247 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:18:28.625815 6162 master_runner.cc:428] Detected that this master 0cc1c8105976413c9be36d2d9f807985 is joining an existing cluster
I20260924 15:18:28.625866 6162 master_runner.cc:432] Initiating AddMaster RPC to add 127.31.12.124:40647
I20260924 15:18:28.633262 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.626917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54478 (local address 127.31.12.126:37657)
0924 15:18:28.626979 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.626982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.627053 (+ 71us) server_negotiation.cc:407] Connection header received
0924 15:18:28.627119 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.627120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.627132 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.627202 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.627718 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.628268 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.629218 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.629560 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.629634 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.630548 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.630550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.630550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.630558 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.632132 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.632508 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.632509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.632510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.632571 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.632749 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.632750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.632750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.632884 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.632956 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.633086 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.633164 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 15:18:28.633312 6253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.626836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.0.0.1:54478)
0924 15:18:28.626993 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.627005 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.627060 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.627325 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.627325 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.627546 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.627634 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.627637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.628349 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.628350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.629128 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.629130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.629647 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.629648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.629705 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.630159 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.630162 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.630462 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.632201 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.632201 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.632202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.632434 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.632636 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.632637 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.632637 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.632688 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.632961 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.632961 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.632993 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.633043 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.633223 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":105,"threads_started":1}
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263107, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.126@KRBTEST.COM for oryx/127.31.12.124@KRBTEST.COM
I20260924 15:18:28.653704 6253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.627254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:40647 (local address 127.0.0.1:53494)
0924 15:18:28.633447 (+ 6193us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.633456 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.633522 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.633656 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.633656 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.633754 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.633791 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.633792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.634485 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.634486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.635187 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.635189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.635501 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.635502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.635556 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.650308 (+ 14752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.650313 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.650716 (+ 403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.652614 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.652615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.652615 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.652826 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.653026 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.653026 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.653027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.653078 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.653334 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.653334 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.653371 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.653433 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.653587 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6182,"thread_start_us":58,"threads_started":1}
I20260924 15:18:28.654151 6233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.627284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53494 (local address 127.31.12.124:40647)
0924 15:18:28.627317 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.627319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.633503 (+ 6184us) server_negotiation.cc:407] Connection header received
0924 15:18:28.633577 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.633578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.633585 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.633608 (+ 23us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.633859 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.634380 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.635255 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.635422 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.635501 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.650802 (+ 15301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.650803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.650803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.650810 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.652521 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.652906 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.652906 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.652907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.652964 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.653139 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.653139 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.653140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.653259 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.653314 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.653939 (+ 625us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.654067 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:18:28.654973 6254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.627090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:35423 (local address 127.0.0.1:43094)
0924 15:18:28.627190 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.627200 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.627236 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.627541 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.627542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.627719 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.627776 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.627778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.628673 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.628674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.629409 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.629410 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.629938 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.629939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.629983 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.650027 (+ 20044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.650032 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.650459 (+ 427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.653405 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.653406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.653406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.654088 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.654303 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.654304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.654304 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.654354 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.654625 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.654626 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.654683 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.654752 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.654868 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":65,"threads_started":1}
I20260924 15:18:28.655016 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.627386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43094 (local address 127.31.12.125:35423)
0924 15:18:28.627429 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.627431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.627439 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:18:28.627458 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.627459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.627468 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.627531 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.627859 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.628589 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.629535 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.629817 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.629884 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.651295 (+ 21411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.651297 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.651297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.651305 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.653328 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.654168 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.654169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.654169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.654228 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.654414 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.654415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.654415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.654555 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.654611 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.654816 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.654893 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:18:28.655778 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 5561
I20260924 15:18:28.656970 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.635349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:40647 (local address 127.0.0.1:53500)
0924 15:18:28.635432 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.635442 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.635474 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.635816 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.635817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.635974 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.636014 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.636015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.636682 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.636683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.637546 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.637549 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.637865 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.637867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.637942 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.638250 (+ 308us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.638255 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.639446 (+ 1191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.655001 (+ 15555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.655002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.655002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.655212 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.655650 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.655651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.655652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.655720 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.656546 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.656547 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.656596 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.656690 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.656819 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 15:18:28.657541 6256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.635537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.31.12.124:40647)
0924 15:18:28.635697 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.635698 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.635703 (+ 5us) server_negotiation.cc:407] Connection header received
0924 15:18:28.635729 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.635730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.635737 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.635786 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.636083 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.636609 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.637614 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.637798 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.637846 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.639550 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.639551 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.639552 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.639557 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.654911 (+ 15354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.655521 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.655522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.655523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.655587 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.655789 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.655791 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.655791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.656437 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.656538 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.657309 (+ 771us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.657384 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":101,"threads_started":1}
I20260924 15:18:28.658188 5499 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.31.12.124:40647: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0cc1c8105976413c9be36d2d9f807985"
member_type: NON_VOTER
last_known_addr {
host: "127.31.12.124"
port: 40647
}
attrs {
promote: true
}
}
dest_uuid: "4b6f5737f73c4374a92ee2c8a69bac93"
cas_config_opid_index: 5
I20260924 15:18:28.658440 5499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: NON_VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: true } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } }
I20260924 15:18:28.658772 6062 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0cc1c8105976413c9be36d2d9f807985 (127.31.12.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: NON_VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: true } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } } }
I20260924 15:18:28.659971 6062 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.31.12.124:40647
I20260924 15:18:28.660039 6062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: NON_VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: true } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } } }
I20260924 15:18:28.660111 6062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:28.660527 6162 master.cc:537] Master@127.31.12.124:40647 shutting down...
I20260924 15:18:28.660465 6204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:28.661338 6204 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985 [term 1 FOLLOWER]: Refusing update from remote peer 4b6f5737f73c4374a92ee2c8a69bac93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 15:18:28.661639 6063 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 15:18:28.662242 5484 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 15:18:28.663210 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:43277
--local_ip_for_outbound_sockets=127.31.12.65
--tserver_master_addrs=127.31.12.126:37657
--webserver_port=32917
--webserver_interface=127.31.12.65
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.65.keytab
--principal=oryx/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.12.126:37657,127.31.12.125:35423
--tserver_master_addrs=127.31.12.126:37657,127.31.12.125:35423,127.31.12.124:40647 with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:18:28.664561 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.660305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:35423 (local address 127.0.0.1:43108)
0924 15:18:28.660666 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.660677 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.660719 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.660901 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.660902 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.660904 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:28.660944 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.660945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.662001 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.662002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.663329 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.663333 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.664257 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.664258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.664330 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.664356 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.664416 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.664465 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346}
I20260924 15:18:28.664842 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.660518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43108 (local address 127.31.12.125:35423)
0924 15:18:28.660638 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.660640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.660765 (+ 125us) server_negotiation.cc:407] Connection header received
0924 15:18:28.660785 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.660786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.660808 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.660855 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:28.661006 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.661808 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.663436 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.664075 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.664144 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.664222 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.664706 (+ 484us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.664760 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 15:18:28.665051 6118 raft_consensus.cc:1261] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 LEARNER]: Refusing update from remote peer 4b6f5737f73c4374a92ee2c8a69bac93: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 15:18:28.665246 6063 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: NON_VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260924 15:18:28.665349 5484 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 -> Peer 0cc1c8105976413c9be36d2d9f807985 (127.31.12.124:40647): Couldn't send request to peer 0cc1c8105976413c9be36d2d9f807985. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 15:18:28.665729 6118 raft_consensus.cc:2932] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0cc1c8105976413c9be36d2d9f807985 (127.31.12.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: NON_VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: true } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } } }
I20260924 15:18:28.666816 6148 sys_catalog.cc:455] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b6f5737f73c4374a92ee2c8a69bac93. Latest consensus state: current_term: 1 leader_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: NON_VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: true } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } } }
I20260924 15:18:28.666930 6148 sys_catalog.cc:458] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [sys.catalog]: This master's current role is: LEARNER
I20260924 15:18:28.666970 6148 sys_catalog.cc:455] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: NON_VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: true } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } } }
I20260924 15:18:28.667021 6148 sys_catalog.cc:458] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [sys.catalog]: This master's current role is: LEARNER
I20260924 15:18:28.668306 6062 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93: attempting to promote NON_VOTER efed9eeea550446e838d24f9dc145216 to VOTER
I20260924 15:18:28.668527 6062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: false } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } }
I20260924 15:18:28.668691 6162 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:18:28.668789 6162 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:28.668833 6162 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985: stopping tablet replica
I20260924 15:18:28.669243 6118 raft_consensus.cc:1261] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 LEARNER]: Refusing update from remote peer 4b6f5737f73c4374a92ee2c8a69bac93: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 15:18:28.669600 6063 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 15:18:28.669983 5484 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 -> Peer 0cc1c8105976413c9be36d2d9f807985 (127.31.12.124:40647): Couldn't send request to peer 0cc1c8105976413c9be36d2d9f807985. Status: Network error: recv got EOF from unknown peer (error 108). This is attempt 1: this message will repeat every 5th retry.
I20260924 15:18:28.670542 6063 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, efed9eeea550446e838d24f9dc145216 (127.31.12.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: false } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } } }
I20260924 15:18:28.671497 6063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: false } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } } }
I20260924 15:18:28.671571 6063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:28.671618 6063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: false } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } } }
I20260924 15:18:28.671650 6063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93 [sys.catalog]: This master's current role is: LEADER
I20260924 15:18:28.671926 6162 master.cc:552] Master@127.31.12.124:40647 shutdown complete.
I20260924 15:18:28.671989 6162 master_runner.cc:305] Clearing existing system tablet
I20260924 15:18:28.672148 6162 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:28.672111 6118 raft_consensus.cc:2932] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, efed9eeea550446e838d24f9dc145216 (127.31.12.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: false } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } } }
I20260924 15:18:28.672210 6162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:28.672963 6260 sys_catalog.cc:455] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: false } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } } }
I20260924 15:18:28.673043 6260 sys_catalog.cc:458] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:18:28.673424 6162 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:18:28.673897 6263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.674028 6162 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:28.674083 6162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.674113 6162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/wal
uuid: "0cc1c8105976413c9be36d2d9f807985"
format_stamp: "Formatted at 2026-09-24 15:18:28 on dist-test-slave-zwbb"
server_key: "78e03bc8b146b74057445681df3fe29b"
server_key_iv: "6877ad06bb51b5fc7aae99dc0166e5a9"
server_key_version: "encryptionkey@0"
I20260924 15:18:28.674674 6162 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:28.675761 6162 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 15:18:28.675843 6162 log.cc:1196] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 15:18:28.676060 6162 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985: Deleting consensus metadata
I20260924 15:18:28.676273 6162 master_runner.cc:315] Copying system tablet from 127.31.12.126:37657
I20260924 15:18:28.676831 6162 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985: tablet copy: Beginning tablet copy session from remote peer at address 127.31.12.126:37657
I20260924 15:18:28.683180 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.677183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54490 (local address 127.31.12.126:37657)
0924 15:18:28.677225 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.677236 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.677351 (+ 115us) server_negotiation.cc:407] Connection header received
0924 15:18:28.677414 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.677415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.677426 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.677495 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.677810 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.678286 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.679026 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.679245 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.679277 (+ 32us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.680271 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.680273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.680273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.680280 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.681885 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.682278 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.682279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.682280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.682335 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.682549 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.682550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.682551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.682663 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.682732 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.682996 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.683077 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:18:28.683184 6269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.677145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.0.0.1:54490)
0924 15:18:28.677276 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.677286 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.677364 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.677505 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.677506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.677687 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.677734 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.677736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.678371 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.678372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.678908 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.678910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.679347 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.679347 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.679399 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.679860 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.679867 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.680163 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.681961 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.681963 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.681963 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.682195 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.682420 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.682421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.682421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.682478 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.682736 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.682737 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.682780 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.682960 (+ 180us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.683082 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260924 15:18:28.683715 5519 tablet_copy_service.cc:140] P 4b6f5737f73c4374a92ee2c8a69bac93: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0cc1c8105976413c9be36d2d9f807985 ({username='oryx', principal='oryx/127.31.12.124@KRBTEST.COM'} at 127.0.0.1:54490)
I20260924 15:18:28.683787 5519 tablet_copy_service.cc:161] P 4b6f5737f73c4374a92ee2c8a69bac93: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0cc1c8105976413c9be36d2d9f807985 at {username='oryx', principal='oryx/127.31.12.124@KRBTEST.COM'} at 127.0.0.1:54490: session id = 0cc1c8105976413c9be36d2d9f807985-00000000000000000000000000000000
I20260924 15:18:28.684207 5519 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 15:18:28.684674 6162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 15:18:28.685664 6162 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985: tablet copy: Starting download of 0 data blocks...
I20260924 15:18:28.685824 6162 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985: tablet copy: Starting download of 1 WAL segments...
I20260924 15:18:28.686517 6162 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 15:18:28.687225 5519 tablet_copy_service.cc:342] P 4b6f5737f73c4374a92ee2c8a69bac93: Request end of tablet copy session 0cc1c8105976413c9be36d2d9f807985-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.31.12.124@KRBTEST.COM'} at 127.0.0.1:54490
I20260924 15:18:28.687294 5519 tablet_copy_service.cc:433] P 4b6f5737f73c4374a92ee2c8a69bac93: ending tablet copy session 0cc1c8105976413c9be36d2d9f807985-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0cc1c8105976413c9be36d2d9f807985
I20260924 15:18:28.689736 6162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:28.690922 6270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:28.694236 6271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:28.697649 6162 server_base.cc:1034] running on GCE node
W20260924 15:18:28.697749 6273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:28.697877 6162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:28.698056 6162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:28.699180 6162 hybrid_clock.cc:648] HybridClock initialized: now 1790263108699173 us; error 27 us; skew 500 ppm
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263108, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:28.700482 6162 init.cc:377] Logged in from keytab as oryx/127.31.12.124@KRBTEST.COM (short username oryx)
I20260924 15:18:28.700776 6162 webserver.cc:460] Webserver started at http://127.31.12.124:35323/ using document root <none> and password file <none>
I20260924 15:18:28.700894 6162 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:28.700946 6162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:28.701546 6162 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:28.702129 6280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.702275 6162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:18:28.702319 6162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/wal
uuid: "0cc1c8105976413c9be36d2d9f807985"
format_stamp: "Formatted at 2026-09-24 15:18:28 on dist-test-slave-zwbb"
server_key: "78e03bc8b146b74057445681df3fe29b"
server_key_iv: "6877ad06bb51b5fc7aae99dc0166e5a9"
server_key_version: "encryptionkey@0"
I20260924 15:18:28.702368 6162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:28.717972 6162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:28.720175 6162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:28.723456 6162 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.124:40647
I20260924 15:18:28.723485 6332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.124:40647 every 8 connection(s)
I20260924 15:18:28.723798 6162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/master-2/data/info.pb
I20260924 15:18:28.729959 6333 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:18:28.730374 6333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985: Bootstrap starting.
I20260924 15:18:28.732681 6333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:18:28.732954 6333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985: Bootstrap complete.
I20260924 15:18:28.733179 6333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: false } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } }
I20260924 15:18:28.733270 6333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0cc1c8105976413c9be36d2d9f807985, State: Initialized, Role: LEARNER
I20260924 15:18:28.733335 6333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: false } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } }
I20260924 15:18:28.733552 6333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:18:28.733544 6336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: false } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } } }
I20260924 15:18:28.733775 6336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985 [sys.catalog]: This master's current role is: LEARNER
I20260924 15:18:28.735462 6348 catalog_manager.cc:1259] Loaded cluster ID: 789d17c9ecf04957b00f86660a385e57
I20260924 15:18:28.735548 6348 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985: loading cluster ID for follower catalog manager: success
I20260924 15:18:28.736276 6348 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985: acquiring CA information for follower catalog manager: success
I20260924 15:18:28.736534 6348 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0cc1c8105976413c9be36d2d9f807985: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 15:18:28.781057 6258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:28.781291 6258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:28.781332 6258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:28.782651 6258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:28.782709 6258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:28.782753 6258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:18:28.784344 6258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.65.keytab
--principal=oryx/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:43277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=32917
--tserver_master_addrs=127.31.12.126:37657,127.31.12.125:35423,127.31.12.124:40647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:28.784597 6258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:28.784825 6258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:28.790670 6354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:28.790692 6356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:28.790658 6353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:28.791076 6258 server_base.cc:1034] running on GCE node
I20260924 15:18:28.791270 6258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:28.791531 6258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:28.792711 6258 hybrid_clock.cc:648] HybridClock initialized: now 1790263108792692 us; error 39 us; skew 500 ppm
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263108, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:28.795128 6258 init.cc:377] Logged in from keytab as oryx/127.31.12.65@KRBTEST.COM (short username oryx)
I20260924 15:18:28.795603 6258 webserver.cc:460] Webserver started at http://127.31.12.65:32917/ using document root <none> and password file <none>
I20260924 15:18:28.795866 6258 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:28.795948 6258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:28.797231 6258 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:28.797866 6363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.798012 6258 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.798107 6258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "27c5c2f9a521468c87e1cbbdcd730d7a"
format_stamp: "Formatted at 2026-09-24 15:18:27 on dist-test-slave-zwbb"
server_key: "9968b7ea2bd37807ae9e50dcc6e23308"
server_key_iv: "9328a74802ed583205ff65032e8245a9"
server_key_version: "encryptionkey@0"
I20260924 15:18:28.798444 6258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:28.813735 6258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:28.816085 6258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:28.816238 6258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:28.816468 6258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:28.816792 6258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:28.816849 6258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.816885 6258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:28.816941 6258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.823310 6258 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:43277
I20260924 15:18:28.823351 6476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:43277 every 8 connection(s)
I20260924 15:18:28.823688 6258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:18:28.825872 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 6258
I20260924 15:18:28.826031 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 5696
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263108, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.125@KRBTEST.COM
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263108, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.124@KRBTEST.COM
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263108, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM
I20260924 15:18:28.836463 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.824373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:38071 (local address 127.31.12.125:35423)
0924 15:18:28.824414 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.824417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.824643 (+ 226us) server_negotiation.cc:407] Connection header received
0924 15:18:28.825489 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.825491 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.825505 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.825556 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.826764 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.827499 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.829071 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.829368 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.829406 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.833293 (+ 3887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.833295 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.833295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.833303 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.835149 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.835627 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.835628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.835628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.835686 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.835890 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.835891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.835891 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.836030 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.836074 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.836282 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.836366 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:18:28.836463 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.824711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:51443 (local address 127.31.12.126:37657)
0924 15:18:28.824753 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.824755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.824991 (+ 236us) server_negotiation.cc:407] Connection header received
0924 15:18:28.825395 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.825396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.825410 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.825518 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.828782 (+ 3264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.829534 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.830488 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.830776 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.830820 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.833294 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.833295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.833295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.833303 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.835284 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.835699 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.835700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.835701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.835757 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.835947 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.835948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.835948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.836075 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.836118 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.836297 (+ 179us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.836368 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:18:28.836668 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.824787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:40647 (local address 127.31.12.65:47791)
0924 15:18:28.825030 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.825052 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.825303 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.825632 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.825633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.826135 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.826418 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.826424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.827744 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.827745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.829004 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.829007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.829626 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.829627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.829694 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.830300 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.830305 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.832644 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.834986 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.834993 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.834995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.835312 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.835519 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.835519 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.835520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.835651 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.835956 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.835963 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.836129 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.836232 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.836412 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":11,"thread_start_us":57,"threads_started":1}
I20260924 15:18:28.836817 6482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.824631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:51443)
0924 15:18:28.824934 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.824945 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.825298 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.825994 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.825995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.828612 (+ 2617us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.828678 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.828680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.829627 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.829627 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.830382 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.830385 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.830904 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.830905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.830976 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.831293 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.831298 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.833190 (+ 1892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.835362 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.835363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.835363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.835618 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.835823 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.835824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.835824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.835875 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.836157 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.836158 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.836187 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.836556 (+ 369us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.836678 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":59,"threads_started":1}
I20260924 15:18:28.836979 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:37031
--local_ip_for_outbound_sockets=127.31.12.66
--tserver_master_addrs=127.31.12.126:37657
--webserver_port=40515
--webserver_interface=127.31.12.66
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.66.keytab
--principal=oryx/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.12.126:37657,127.31.12.125:35423
--tserver_master_addrs=127.31.12.126:37657,127.31.12.125:35423,127.31.12.124:40647 with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:18:28.837127 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.824872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:47791 (local address 127.31.12.124:40647)
0924 15:18:28.824993 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.824995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.825094 (+ 99us) server_negotiation.cc:407] Connection header received
0924 15:18:28.825486 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.825488 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.825502 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.825566 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.826526 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.827649 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.829104 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.829508 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.829570 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.832743 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.832744 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.832745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.832752 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.834890 (+ 2138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.835393 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.835394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.835395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.835455 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.835715 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.835716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.835716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.835861 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.835909 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.836901 (+ 992us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.837004 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":80,"threads_started":1}
I20260924 15:18:28.837347 6481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.824346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:35423 (local address 127.31.12.65:38071)
0924 15:18:28.824503 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.824554 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.825294 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.825613 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.825617 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.826568 (+ 951us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.826639 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.826640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.827639 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.827640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.828966 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.828968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.829516 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.829517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.829662 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.830264 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.830274 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.833190 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.835262 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.835263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.835264 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.835529 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.835757 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.835758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.835758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.835817 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.836122 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.836123 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.836164 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.837206 (+ 1042us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.837284 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":82,"thread_start_us":89,"threads_started":1}
I20260924 15:18:28.837913 6479 heartbeater.cc:347] Connected to a master server at 127.31.12.125:35423
I20260924 15:18:28.837981 6479 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:28.838174 6479 heartbeater.cc:511] Master 127.31.12.125:35423 requested a full tablet report, sending...
I20260924 15:18:28.838733 6478 heartbeater.cc:347] Connected to a master server at 127.31.12.126:37657
I20260924 15:18:28.838749 6108 ts_manager.cc:194] Registered new tserver with Master: 27c5c2f9a521468c87e1cbbdcd730d7a (127.31.12.65:43277)
I20260924 15:18:28.838788 6478 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:28.838861 6478 heartbeater.cc:511] Master 127.31.12.126:37657 requested a full tablet report, sending...
I20260924 15:18:28.839092 6477 heartbeater.cc:347] Connected to a master server at 127.31.12.124:40647
I20260924 15:18:28.839145 6477 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:28.839195 5499 ts_manager.cc:194] Re-registered known tserver with Master: 27c5c2f9a521468c87e1cbbdcd730d7a (127.31.12.65:43277)
I20260924 15:18:28.839217 6477 heartbeater.cc:511] Master 127.31.12.124:40647 requested a full tablet report, sending...
I20260924 15:18:28.839562 6288 ts_manager.cc:194] Registered new tserver with Master: 27c5c2f9a521468c87e1cbbdcd730d7a (127.31.12.65:43277)
I20260924 15:18:28.839619 5499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:51443
W20260924 15:18:28.917088 6485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:28.917272 6485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:28.917295 6485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:28.918367 6485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:28.918403 6485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:28.918435 6485 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:18:28.919770 6485 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.66.keytab
--principal=oryx/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:37031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=40515
--tserver_master_addrs=127.31.12.126:37657,127.31.12.125:35423,127.31.12.124:40647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:28.919931 6485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:28.920147 6485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:18:28.925371 6485 server_base.cc:1034] running on GCE node
W20260924 15:18:28.925335 6491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:28.925292 6493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:28.925288 6490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:28.925797 6485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:28.926079 6485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:28.927233 6485 hybrid_clock.cc:648] HybridClock initialized: now 1790263108927222 us; error 23 us; skew 500 ppm
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263108, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:28.929627 6485 init.cc:377] Logged in from keytab as oryx/127.31.12.66@KRBTEST.COM (short username oryx)
I20260924 15:18:28.930058 6485 webserver.cc:460] Webserver started at http://127.31.12.66:40515/ using document root <none> and password file <none>
I20260924 15:18:28.930279 6485 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:28.930358 6485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:28.931644 6485 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:28.932242 6500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.932443 6485 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:18:28.932528 6485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "eb0abe6f3cda4468b8e17438d5d184bd"
format_stamp: "Formatted at 2026-09-24 15:18:27 on dist-test-slave-zwbb"
server_key: "752fbc5387dbaa59a8e85a2802ab9e3b"
server_key_iv: "9561bf3329b8a324d41a02724c620bdb"
server_key_version: "encryptionkey@0"
I20260924 15:18:28.932859 6485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:28.956493 6485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:28.958446 6485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:28.958564 6485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:28.958751 6485 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:28.959090 6485 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:28.959144 6485 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.959183 6485 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:28.959209 6485 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:28.964972 6485 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:37031
I20260924 15:18:28.965046 6614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:37031 every 8 connection(s)
I20260924 15:18:28.965377 6485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263108, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.125@KRBTEST.COM
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263108, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM
Sep 24 15:18:28 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263108, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.124@KRBTEST.COM
I20260924 15:18:28.972805 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 6485
I20260924 15:18:28.972968 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 5832
I20260924 15:18:28.976269 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.966239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:35947 (local address 127.31.12.126:37657)
0924 15:18:28.966317 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.966319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.966539 (+ 220us) server_negotiation.cc:407] Connection header received
0924 15:18:28.967130 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.967131 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.967145 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.967196 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.968023 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.968772 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.969839 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.970124 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.970178 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.973017 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.973019 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.973019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.973028 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.974839 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.975246 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.975248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.975248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.975305 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.975545 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.975546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.975546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.975662 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.975731 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.976044 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.976129 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 15:18:28.976289 6620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.966211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:35947)
0924 15:18:28.966386 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.966419 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.967020 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.967261 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.967265 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.967728 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.967912 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.967918 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.968866 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.968867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.969769 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.969771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.970198 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.970199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.970265 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.970874 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.970878 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.972891 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.974939 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.974946 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.974948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.975157 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.975369 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.975370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.975370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.975477 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.975734 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.975741 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.975885 (+ 144us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.975995 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.976141 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":49,"thread_start_us":104,"threads_started":1}
I20260924 15:18:28.976766 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.966723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:44165 (local address 127.31.12.125:35423)
0924 15:18:28.966792 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.966794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.966909 (+ 115us) server_negotiation.cc:407] Connection header received
0924 15:18:28.967316 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.967318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.967328 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.967355 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.968006 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.968494 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.969468 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.969666 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.969742 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.972161 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.972163 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.972164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.972171 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.975180 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.975570 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.975572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.975572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.975625 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.975808 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.975809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.975809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.975939 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.976003 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.976548 (+ 545us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.976633 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 15:18:28.977137 6622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.966669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:35423 (local address 127.31.12.66:44165)
0924 15:18:28.966837 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.966846 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.967227 (+ 381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.967414 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.967414 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.967709 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.967911 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.967915 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.968604 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.968604 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.969356 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.969357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.969749 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.969750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.969880 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.970374 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.970382 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.972067 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.975266 (+ 3199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.975267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.975268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.975493 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.975692 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.975692 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.975693 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.975748 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.976369 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.976370 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.976414 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.976876 (+ 462us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.976985 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
I20260924 15:18:28.977593 6618 heartbeater.cc:347] Connected to a master server at 127.31.12.125:35423
I20260924 15:18:28.977698 6618 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:28.977902 6618 heartbeater.cc:511] Master 127.31.12.125:35423 requested a full tablet report, sending...
I20260924 15:18:28.978147 6617 heartbeater.cc:347] Connected to a master server at 127.31.12.126:37657
I20260924 15:18:28.978245 6617 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:28.978353 6617 heartbeater.cc:511] Master 127.31.12.126:37657 requested a full tablet report, sending...
I20260924 15:18:28.978384 6108 ts_manager.cc:194] Registered new tserver with Master: eb0abe6f3cda4468b8e17438d5d184bd (127.31.12.66:37031)
I20260924 15:18:28.978722 5499 ts_manager.cc:194] Re-registered known tserver with Master: eb0abe6f3cda4468b8e17438d5d184bd (127.31.12.66:37031)
I20260924 15:18:28.979246 5499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:35947
I20260924 15:18:28.979758 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:32865
--local_ip_for_outbound_sockets=127.31.12.67
--tserver_master_addrs=127.31.12.126:37657
--webserver_port=45911
--webserver_interface=127.31.12.67
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.67.keytab
--principal=oryx/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:45129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.31.12.126:37657,127.31.12.125:35423
--tserver_master_addrs=127.31.12.126:37657,127.31.12.125:35423,127.31.12.124:40647 with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:18:28.980616 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.966564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:32935 (local address 127.31.12.124:40647)
0924 15:18:28.966602 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:28.966604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:28.966755 (+ 151us) server_negotiation.cc:407] Connection header received
0924 15:18:28.967193 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:28.967195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:28.967208 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:28.967265 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:28.968023 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.968773 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.970250 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.970577 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.970648 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.977797 (+ 7149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:28.977798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:28.977798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:28.977806 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:28.979477 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.979837 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.979838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.979839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.979899 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:28.980115 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:28.980116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:28.980116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:28.980247 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:28.980287 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:28.980482 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 15:18:28.980559 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:18:28.980707 6621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:28.966474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:40647 (local address 127.31.12.66:32935)
0924 15:18:28.966605 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:28.966616 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:28.967037 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:28.967267 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:28.967268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:28.967712 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:28.967912 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.967918 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.968866 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.968867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.969770 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:28.969771 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:28.970659 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:28.970660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:28.970715 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:28.971064 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:28.971069 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:28.972960 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:28.979555 (+ 6595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.979556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.979556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.979752 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.979987 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:28.979988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:28.979989 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:28.980048 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:28.980331 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:28.980332 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:28.980372 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 15:18:28.980438 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 15:18:28.980555 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":56,"thread_start_us":90,"threads_started":1}
I20260924 15:18:28.981235 6616 heartbeater.cc:347] Connected to a master server at 127.31.12.124:40647
I20260924 15:18:28.981294 6616 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:28.981359 6616 heartbeater.cc:511] Master 127.31.12.124:40647 requested a full tablet report, sending...
I20260924 15:18:28.981683 6288 ts_manager.cc:194] Registered new tserver with Master: eb0abe6f3cda4468b8e17438d5d184bd (127.31.12.66:37031)
W20260924 15:18:29.060738 6623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:18:29.060940 6623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:18:29.060963 6623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:18:29.062187 6623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:18:29.062229 6623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:18:29.062279 6623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:18:29.063722 6623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:45129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.67.keytab
--principal=oryx/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:32865
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=45911
--tserver_master_addrs=127.31.12.126:37657,127.31.12.125:35423,127.31.12.124:40647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:18:29.063963 6623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:18:29.064222 6623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:18:29.069355 6631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:29.069406 6629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:18:29.069372 6628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:18:29.069555 6623 server_base.cc:1034] running on GCE node
I20260924 15:18:29.069868 6623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:18:29.070123 6623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:18:29.071297 6623 hybrid_clock.cc:648] HybridClock initialized: now 1790263109071285 us; error 28 us; skew 500 ppm
Sep 24 15:18:29 dist-test-slave-zwbb krb5kdc[5453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263109, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:18:29.073714 6623 init.cc:377] Logged in from keytab as oryx/127.31.12.67@KRBTEST.COM (short username oryx)
I20260924 15:18:29.074152 6623 webserver.cc:460] Webserver started at http://127.31.12.67:45911/ using document root <none> and password file <none>
I20260924 15:18:29.074450 6623 fs_manager.cc:359] Metadata directory not provided
I20260924 15:18:29.074506 6623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:18:29.075766 6623 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:18:29.076301 6638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:29.076500 6623 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:29.076576 6623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "a2dfc048810b439d880bf058dfda0a06"
format_stamp: "Formatted at 2026-09-24 15:18:28 on dist-test-slave-zwbb"
server_key: "b6853d24b905e98d3bfe4ddbc55f724d"
server_key_iv: "d912a5cf5cc7a208955a9f4912f13fca"
server_key_version: "encryptionkey@0"
I20260924 15:18:29.076881 6623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:18:29.088066 6623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:18:29.090242 6623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:18:29.090380 6623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:18:29.090552 6623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:18:29.090847 6623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:18:29.090879 6623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:29.090937 6623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:18:29.090963 6623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:18:29.096583 6623 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:32865
I20260924 15:18:29.096614 6752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:32865 every 8 connection(s)
I20260924 15:18:29.096989 6623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:18:29 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263109, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.124@KRBTEST.COM
Sep 24 15:18:29 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263109, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.125@KRBTEST.COM
I20260924 15:18:29.104903 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 6623
Sep 24 15:18:29 dist-test-slave-zwbb krb5kdc[5453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263109, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM
I20260924 15:18:29.110066 6757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.097708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:43365)
0924 15:18:29.097879 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:29.097917 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:29.098564 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:29.098831 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:29.098835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:29.099146 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:29.099315 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:29.099320 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.100222 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.100223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:29.101131 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:29.101133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.101684 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.101684 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:29.101747 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.103257 (+ 1510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:29.103261 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:29.106061 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:29.108034 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:29.108040 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:29.108043 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:29.108358 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:29.108574 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:29.108575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:29.108575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:29.108730 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:29.109353 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:29.109360 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:29.109543 (+ 183us) client_negotiation.cc:769] Sending connection context
0924 15:18:29.109653 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 15:18:29.109830 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":62,"thread_start_us":88,"threads_started":1}
I20260924 15:18:29.110178 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.098268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:37229 (local address 127.31.12.124:40647)
0924 15:18:29.098332 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:29.098334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:29.098342 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:18:29.098680 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:29.098681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:29.098693 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:29.098723 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:29.099405 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.099890 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.100868 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.101106 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.101168 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.106564 (+ 5396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:29.106566 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:29.106566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:29.106575 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:29.108460 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:29.108946 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:29.108947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:29.108947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:29.109006 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:29.109568 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:29.109569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:29.109570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:29.109717 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:29.109767 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:29.109986 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 15:18:29.110078 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 15:18:29.110360 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.097726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:43365 (local address 127.31.12.126:37657)
0924 15:18:29.097787 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:29.097789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:29.098009 (+ 220us) server_negotiation.cc:407] Connection header received
0924 15:18:29.098683 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:29.098685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:29.098698 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:29.098727 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:29.099418 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.100139 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.101217 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.101538 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.101580 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.106162 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:29.106164 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:29.106164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:29.106172 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:29.107942 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:29.108445 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:29.108446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:29.108446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:29.108503 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:29.108802 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:29.108803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:29.108803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:29.108971 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:29.109030 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:29.110188 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0924 15:18:29.110271 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:18:29.110404 6758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.097945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:35423 (local address 127.31.12.67:60647)
0924 15:18:29.098072 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:29.098123 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:29.098566 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:29.098831 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:29.098835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:29.099235 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:29.099315 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:29.099320 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.100222 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.100223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:29.101103 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:29.101106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.101684 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.101685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:29.102235 (+ 550us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.102915 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:29.102920 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:29.105305 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:29.109058 (+ 3753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:29.109060 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:29.109060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:29.109281 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:29.109726 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:29.109727 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:29.109727 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:29.109786 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:29.110088 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:29.110088 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:29.110145 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 15:18:29.110209 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 15:18:29.110295 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":27,"thread_start_us":69,"threads_started":1}
I20260924 15:18:29.110669 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.098062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:60647 (local address 127.31.12.125:35423)
0924 15:18:29.098105 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:29.098107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:29.098188 (+ 81us) server_negotiation.cc:407] Connection header received
0924 15:18:29.098680 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:29.098681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:29.098692 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:29.098742 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:29.099427 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.100131 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.101204 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.101538 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.101580 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.105819 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:29.105821 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:29.105821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:29.105829 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:29.108800 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:29.109365 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:29.109366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:29.109367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:29.109461 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:29.109859 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:29.109860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:29.109860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:29.109985 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:29.110034 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:29.110500 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 15:18:29.110577 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:18:29.111833 6755 heartbeater.cc:347] Connected to a master server at 127.31.12.125:35423
I20260924 15:18:29.111931 6755 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:29.112093 6754 heartbeater.cc:347] Connected to a master server at 127.31.12.126:37657
I20260924 15:18:29.112128 6755 heartbeater.cc:511] Master 127.31.12.125:35423 requested a full tablet report, sending...
I20260924 15:18:29.112145 6754 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:29.112228 6754 heartbeater.cc:511] Master 127.31.12.126:37657 requested a full tablet report, sending...
I20260924 15:18:29.112663 6759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.098096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:40647 (local address 127.31.12.67:37229)
0924 15:18:29.098225 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:29.098233 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:29.098574 (+ 341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:29.098831 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:29.098835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:29.099133 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:18:29.099315 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:29.099319 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.099992 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.099992 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:29.100784 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:29.100787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.101269 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.101271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:29.101518 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.102440 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:18:29.102450 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:18:29.106216 (+ 3766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:18:29.108573 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:29.108574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:29.108574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:29.108855 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:29.109378 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:18:29.109379 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:18:29.109380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:18:29.109478 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:18:29.109820 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:18:29.109821 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:18:29.109864 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 15:18:29.112423 (+ 2559us) client_negotiation.cc:240] Negotiation successful
0924 15:18:29.112534 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260924 15:18:29.112658 6108 ts_manager.cc:194] Registered new tserver with Master: a2dfc048810b439d880bf058dfda0a06 (127.31.12.67:32865)
I20260924 15:18:29.112843 5499 ts_manager.cc:194] Re-registered known tserver with Master: a2dfc048810b439d880bf058dfda0a06 (127.31.12.67:32865)
I20260924 15:18:29.113174 6753 heartbeater.cc:347] Connected to a master server at 127.31.12.124:40647
I20260924 15:18:29.113217 6753 heartbeater.cc:464] Registering TS with master...
I20260924 15:18:29.113301 5499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:43365
I20260924 15:18:29.113394 6753 heartbeater.cc:511] Master 127.31.12.124:40647 requested a full tablet report, sending...
I20260924 15:18:29.113866 6288 ts_manager.cc:194] Registered new tserver with Master: a2dfc048810b439d880bf058dfda0a06 (127.31.12.67:32865)
I20260924 15:18:29.119884 5539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.109126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54492 (local address 127.31.12.126:37657)
0924 15:18:29.110438 (+ 1312us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:29.110440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:29.111008 (+ 568us) server_negotiation.cc:407] Connection header received
0924 15:18:29.111119 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:29.111120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:29.111131 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:29.111919 (+ 788us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:29.112674 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.113371 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.114407 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.114683 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.114721 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.116575 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:29.116602 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:29.116602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:29.116609 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:29.118564 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:29.119022 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:29.119023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:29.119024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:29.119080 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:29.119290 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:29.119291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:29.119291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:29.119420 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:29.119496 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:29.119705 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 15:18:29.119784 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1296,"thread_start_us":176,"threads_started":1}
I20260924 15:18:29.120179 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.109643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43116 (local address 127.31.12.125:35423)
0924 15:18:29.110765 (+ 1122us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:29.110767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:29.111420 (+ 653us) server_negotiation.cc:407] Connection header received
0924 15:18:29.111459 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:29.111460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:29.111470 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:29.111516 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:29.112509 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.113475 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.114501 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.114785 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.114840 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.116514 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:29.116515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:29.116516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:29.116523 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:29.118720 (+ 2197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:29.119131 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:29.119132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:29.119133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:29.119186 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:29.119378 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:29.119379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:29.119379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:29.119509 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:29.119572 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:29.119768 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 15:18:29.119832 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1106,"thread_start_us":51,"threads_started":1}
I20260924 15:18:29.122391 5499 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:18:29.124686 5499 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:18:29.126518 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.111767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53514 (local address 127.31.12.124:40647)
0924 15:18:29.112404 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:29.112406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:29.114302 (+ 1896us) server_negotiation.cc:407] Connection header received
0924 15:18:29.114333 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:29.114333 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:29.114344 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:29.114415 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:29.115322 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.116014 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.117141 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.117559 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.117623 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.119548 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:29.119550 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:29.119550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:29.119559 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:29.122149 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:29.125537 (+ 3388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:29.125538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:29.125538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:29.125602 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:29.125821 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:29.125822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:29.125822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:29.125983 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:29.126105 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:29.126318 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 15:18:29.126405 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613}
I20260924 15:18:29.132901 6061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.128524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:43277 (local address 127.0.0.1:38930)
0924 15:18:29.128835 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:29.128847 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:29.128893 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:29.129515 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:29.129516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:29.129518 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:29.129560 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:29.129561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.130858 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.130859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:29.131768 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:29.131771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.132662 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.132663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:29.132726 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.132729 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:18:29.132791 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 15:18:29.132832 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":46}
I20260924 15:18:29.132931 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.128838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38930 (local address 127.31.12.65:43277)
0924 15:18:29.129278 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:29.129283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:29.129294 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:18:29.129338 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:29.129339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:29.129356 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:29.129637 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:29.129698 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.130748 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.131854 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.132542 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.132652 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.132730 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:29.132806 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 15:18:29.132842 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":66,"threads_started":1}
I20260924 15:18:29.133687 6773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.128622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:32865 (local address 127.0.0.1:36360)
0924 15:18:29.129175 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:29.129184 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:29.129235 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:29.129533 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:29.129533 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:29.129535 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:29.129565 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:29.129566 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.130833 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.130834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:29.132084 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:29.132087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.133367 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.133368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:29.133467 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.133472 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:18:29.133524 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 15:18:29.133563 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":27,"thread_start_us":72,"threads_started":1}
I20260924 15:18:29.133749 6776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.128958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.31.12.66:37031)
0924 15:18:29.129142 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:29.129148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:29.129664 (+ 516us) server_negotiation.cc:407] Connection header received
0924 15:18:29.129752 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:29.129754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:29.129770 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:29.129846 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:29.129973 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.131120 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.132339 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.133188 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.133253 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.133322 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:29.133622 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 15:18:29.133661 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260924 15:18:29.133872 6774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.128741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:37031 (local address 127.0.0.1:51048)
0924 15:18:29.129594 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:29.129605 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:29.129669 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:29.129854 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:29.129855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:29.129857 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:29.129913 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:29.129915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.131211 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.131212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:29.132255 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:29.132257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.133360 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.133361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:29.133425 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.133429 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:18:29.133515 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 15:18:29.133557 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"thread_start_us":74,"threads_started":1}
I20260924 15:18:29.133805 6411 tablet_service.cc:1467] Processing CreateTablet for tablet bcbeb038ca9a45aca857b81d98ebca81 (DEFAULT_TABLE table=test-table [id=82876e91f9634d158c7c4a4090ec05de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:29.134217 6411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcbeb038ca9a45aca857b81d98ebca81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:29.134284 6777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.129051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36360 (local address 127.31.12.67:32865)
0924 15:18:29.129311 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:29.129314 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:29.129322 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:18:29.129355 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:29.129357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:29.129371 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:29.129469 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:29.129641 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.130756 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.132170 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.133282 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.133354 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.133433 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:29.133538 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 15:18:29.133577 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":72,"threads_started":1}
I20260924 15:18:29.134965 6686 tablet_service.cc:1467] Processing CreateTablet for tablet bcbeb038ca9a45aca857b81d98ebca81 (DEFAULT_TABLE table=test-table [id=82876e91f9634d158c7c4a4090ec05de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:29.135330 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcbeb038ca9a45aca857b81d98ebca81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:29.137158 6778 tablet_bootstrap.cc:492] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a: Bootstrap starting.
I20260924 15:18:29.137850 6548 tablet_service.cc:1467] Processing CreateTablet for tablet bcbeb038ca9a45aca857b81d98ebca81 (DEFAULT_TABLE table=test-table [id=82876e91f9634d158c7c4a4090ec05de]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:18:29.137990 6778 tablet_bootstrap.cc:654] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:29.138187 6548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcbeb038ca9a45aca857b81d98ebca81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:18:29.138307 6778 log.cc:824] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:29.140873 6781 tablet_bootstrap.cc:492] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd: Bootstrap starting.
I20260924 15:18:29.141083 6780 tablet_bootstrap.cc:492] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06: Bootstrap starting.
I20260924 15:18:29.141599 6781 tablet_bootstrap.cc:654] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:29.141688 6780 tablet_bootstrap.cc:654] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06: Neither blocks nor log segments found. Creating new log.
I20260924 15:18:29.141925 6781 log.cc:824] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:29.141978 6780 log.cc:824] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06: Log is configured to *not* fsync() on all Append() calls
I20260924 15:18:29.142089 6778 tablet_bootstrap.cc:492] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a: No bootstrap required, opened a new log
I20260924 15:18:29.142202 6778 ts_tablet_manager.cc:1397] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 15:18:29.145257 6778 raft_consensus.cc:348] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27c5c2f9a521468c87e1cbbdcd730d7a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43277 } } peers { permanent_uuid: "a2dfc048810b439d880bf058dfda0a06" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32865 } } peers { permanent_uuid: "eb0abe6f3cda4468b8e17438d5d184bd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 37031 } }
I20260924 15:18:29.145470 6778 raft_consensus.cc:374] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:29.145536 6778 raft_consensus.cc:729] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27c5c2f9a521468c87e1cbbdcd730d7a, State: Initialized, Role: FOLLOWER
I20260924 15:18:29.145661 6778 consensus_queue.cc:260] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27c5c2f9a521468c87e1cbbdcd730d7a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43277 } } peers { permanent_uuid: "a2dfc048810b439d880bf058dfda0a06" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32865 } } peers { permanent_uuid: "eb0abe6f3cda4468b8e17438d5d184bd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 37031 } }
I20260924 15:18:29.145968 6778 ts_tablet_manager.cc:1428] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260924 15:18:29.146241 6781 tablet_bootstrap.cc:492] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd: No bootstrap required, opened a new log
I20260924 15:18:29.146322 6478 heartbeater.cc:503] Master 127.31.12.126:37657 was elected leader, sending a full tablet report...
I20260924 15:18:29.146348 6781 ts_tablet_manager.cc:1397] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260924 15:18:29.149202 6781 raft_consensus.cc:348] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27c5c2f9a521468c87e1cbbdcd730d7a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43277 } } peers { permanent_uuid: "a2dfc048810b439d880bf058dfda0a06" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32865 } } peers { permanent_uuid: "eb0abe6f3cda4468b8e17438d5d184bd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 37031 } }
I20260924 15:18:29.149407 6781 raft_consensus.cc:374] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:29.149479 6781 raft_consensus.cc:729] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb0abe6f3cda4468b8e17438d5d184bd, State: Initialized, Role: FOLLOWER
I20260924 15:18:29.149638 6781 consensus_queue.cc:260] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27c5c2f9a521468c87e1cbbdcd730d7a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43277 } } peers { permanent_uuid: "a2dfc048810b439d880bf058dfda0a06" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32865 } } peers { permanent_uuid: "eb0abe6f3cda4468b8e17438d5d184bd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 37031 } }
I20260924 15:18:29.150151 6617 heartbeater.cc:503] Master 127.31.12.126:37657 was elected leader, sending a full tablet report...
I20260924 15:18:29.149911 6781 ts_tablet_manager.cc:1428] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260924 15:18:29.151234 6780 tablet_bootstrap.cc:492] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06: No bootstrap required, opened a new log
I20260924 15:18:29.151367 6780 ts_tablet_manager.cc:1397] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260924 15:18:29.152776 6780 raft_consensus.cc:348] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27c5c2f9a521468c87e1cbbdcd730d7a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43277 } } peers { permanent_uuid: "a2dfc048810b439d880bf058dfda0a06" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32865 } } peers { permanent_uuid: "eb0abe6f3cda4468b8e17438d5d184bd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 37031 } }
I20260924 15:18:29.152946 6780 raft_consensus.cc:374] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:18:29.152992 6780 raft_consensus.cc:729] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2dfc048810b439d880bf058dfda0a06, State: Initialized, Role: FOLLOWER
I20260924 15:18:29.153138 6780 consensus_queue.cc:260] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27c5c2f9a521468c87e1cbbdcd730d7a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43277 } } peers { permanent_uuid: "a2dfc048810b439d880bf058dfda0a06" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32865 } } peers { permanent_uuid: "eb0abe6f3cda4468b8e17438d5d184bd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 37031 } }
I20260924 15:18:29.153398 6780 ts_tablet_manager.cc:1428] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:18:29.153473 6754 heartbeater.cc:503] Master 127.31.12.126:37657 was elected leader, sending a full tablet report...
I20260924 15:18:29.168906 6785 raft_consensus.cc:482] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:18:29.168998 6785 raft_consensus.cc:504] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27c5c2f9a521468c87e1cbbdcd730d7a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43277 } } peers { permanent_uuid: "a2dfc048810b439d880bf058dfda0a06" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32865 } } peers { permanent_uuid: "eb0abe6f3cda4468b8e17438d5d184bd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 37031 } }
I20260924 15:18:29.169375 6785 leader_election.cc:288] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27c5c2f9a521468c87e1cbbdcd730d7a (127.31.12.65:43277), a2dfc048810b439d880bf058dfda0a06 (127.31.12.67:32865)
I20260924 15:18:29.172171 6777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.169733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:38133 (local address 127.31.12.67:32865)
0924 15:18:29.169781 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:29.169784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:29.169791 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:18:29.169814 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:29.169814 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:29.169822 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:29.169898 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:29.170055 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.170519 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.171464 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.171816 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.171897 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.171905 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:29.172070 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 15:18:29.172106 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 15:18:29.172173 6622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.169650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:32865 (local address 127.31.12.66:38133)
0924 15:18:29.169695 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:29.169705 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:29.169741 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:29.169901 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:29.169903 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:29.169904 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:29.169980 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:29.169981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.170616 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.170617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:29.171368 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:29.171370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.171894 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.171895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:29.171979 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.171982 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:18:29.172062 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 15:18:29.172106 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27,"mutex_wait_us":18}
I20260924 15:18:29.172245 6775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.169614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:44791 (local address 127.31.12.65:43277)
0924 15:18:29.169661 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:29.169663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:29.169692 (+ 29us) server_negotiation.cc:407] Connection header received
0924 15:18:29.169846 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:29.169847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:29.169855 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:29.169943 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:18:29.170092 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.170562 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.171458 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.171882 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.171970 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.171978 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:29.172137 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 15:18:29.172174 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:18:29.172305 6621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.169536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:43277 (local address 127.31.12.66:44791)
0924 15:18:29.169642 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:29.169655 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:18:29.169709 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:18:29.169949 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:18:29.169950 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:18:29.169952 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:18:29.170015 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:29.170017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.170656 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.170657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:29.171367 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:18:29.171369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.171968 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.171969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:18:29.172055 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.172058 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:18:29.172130 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 15:18:29.172173 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 15:18:29.172534 6706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbeb038ca9a45aca857b81d98ebca81" candidate_uuid: "eb0abe6f3cda4468b8e17438d5d184bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2dfc048810b439d880bf058dfda0a06" is_pre_election: true
I20260924 15:18:29.172595 6431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbeb038ca9a45aca857b81d98ebca81" candidate_uuid: "eb0abe6f3cda4468b8e17438d5d184bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27c5c2f9a521468c87e1cbbdcd730d7a" is_pre_election: true
I20260924 15:18:29.172678 6706 raft_consensus.cc:2454] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb0abe6f3cda4468b8e17438d5d184bd in term 0.
I20260924 15:18:29.172741 6431 raft_consensus.cc:2454] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb0abe6f3cda4468b8e17438d5d184bd in term 0.
I20260924 15:18:29.172919 6503 leader_election.cc:302] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2dfc048810b439d880bf058dfda0a06, eb0abe6f3cda4468b8e17438d5d184bd; no voters:
I20260924 15:18:29.173076 6785 raft_consensus.cc:2781] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:18:29.173146 6785 raft_consensus.cc:482] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:18:29.173171 6785 raft_consensus.cc:3037] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:29.174057 6785 raft_consensus.cc:504] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27c5c2f9a521468c87e1cbbdcd730d7a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43277 } } peers { permanent_uuid: "a2dfc048810b439d880bf058dfda0a06" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32865 } } peers { permanent_uuid: "eb0abe6f3cda4468b8e17438d5d184bd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 37031 } }
I20260924 15:18:29.174187 6785 leader_election.cc:288] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [CANDIDATE]: Term 1 election: Requested vote from peers 27c5c2f9a521468c87e1cbbdcd730d7a (127.31.12.65:43277), a2dfc048810b439d880bf058dfda0a06 (127.31.12.67:32865)
I20260924 15:18:29.174337 6431 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbeb038ca9a45aca857b81d98ebca81" candidate_uuid: "eb0abe6f3cda4468b8e17438d5d184bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27c5c2f9a521468c87e1cbbdcd730d7a"
I20260924 15:18:29.174351 6706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcbeb038ca9a45aca857b81d98ebca81" candidate_uuid: "eb0abe6f3cda4468b8e17438d5d184bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2dfc048810b439d880bf058dfda0a06"
I20260924 15:18:29.174417 6431 raft_consensus.cc:3037] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:29.174440 6706 raft_consensus.cc:3037] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:18:29.175117 6431 raft_consensus.cc:2454] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb0abe6f3cda4468b8e17438d5d184bd in term 1.
I20260924 15:18:29.175268 6503 leader_election.cc:302] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27c5c2f9a521468c87e1cbbdcd730d7a, eb0abe6f3cda4468b8e17438d5d184bd; no voters:
I20260924 15:18:29.175345 6785 raft_consensus.cc:2781] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:18:29.175500 6785 raft_consensus.cc:686] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [term 1 LEADER]: Becoming Leader. State: Replica: eb0abe6f3cda4468b8e17438d5d184bd, State: Running, Role: LEADER
I20260924 15:18:29.175589 6785 consensus_queue.cc:237] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27c5c2f9a521468c87e1cbbdcd730d7a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43277 } } peers { permanent_uuid: "a2dfc048810b439d880bf058dfda0a06" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32865 } } peers { permanent_uuid: "eb0abe6f3cda4468b8e17438d5d184bd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 37031 } }
I20260924 15:18:29.175653 6706 raft_consensus.cc:2454] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb0abe6f3cda4468b8e17438d5d184bd in term 1.
I20260924 15:18:29.177021 5499 catalog_manager.cc:5495] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd reported cstate change: term changed from 0 to 1, leader changed from <none> to eb0abe6f3cda4468b8e17438d5d184bd (127.31.12.66). New cstate: current_term: 1 leader_uuid: "eb0abe6f3cda4468b8e17438d5d184bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27c5c2f9a521468c87e1cbbdcd730d7a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43277 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2dfc048810b439d880bf058dfda0a06" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb0abe6f3cda4468b8e17438d5d184bd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 37031 } health_report { overall_health: HEALTHY } } }
I20260924 15:18:29.193262 6776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.191159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51058 (local address 127.31.12.66:37031)
0924 15:18:29.191216 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:29.191218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:29.191226 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:18:29.191278 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:29.191279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:29.191294 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:29.191350 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:18:29.191462 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.191905 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.192613 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.192787 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.192836 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.193002 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:18:29.193102 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:18:29.193162 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:29.193193 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 15:18:29.193210 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 15:18:29.195041 6706 raft_consensus.cc:1261] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06 [term 1 FOLLOWER]: Refusing update from remote peer eb0abe6f3cda4468b8e17438d5d184bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:29.195076 6431 raft_consensus.cc:1261] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a [term 1 FOLLOWER]: Refusing update from remote peer eb0abe6f3cda4468b8e17438d5d184bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:18:29.195428 6785 consensus_queue.cc:1035] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2dfc048810b439d880bf058dfda0a06" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:29.195597 6787 consensus_queue.cc:1035] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "27c5c2f9a521468c87e1cbbdcd730d7a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:18:29.198863 6792 mvcc.cc:205] Tried to move back new op lower bound from 7332917695260291072 to 7332917695184130048. Current Snapshot: MvccSnapshot[applied={T|T < 7332917695260291072}]
I20260924 15:18:29.200093 5498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:18:29.200207 5498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:18:29.202028 5498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b6f5737f73c4374a92ee2c8a69bac93: Sending DeleteTablet for 3 replicas of tablet bcbeb038ca9a45aca857b81d98ebca81
I20260924 15:18:29.202361 6686 tablet_service.cc:1514] Processing DeleteTablet for tablet bcbeb038ca9a45aca857b81d98ebca81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:29 UTC) from {username='oryx', principal='oryx/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:36360
I20260924 15:18:29.202425 6548 tablet_service.cc:1514] Processing DeleteTablet for tablet bcbeb038ca9a45aca857b81d98ebca81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:29 UTC) from {username='oryx', principal='oryx/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:51048
I20260924 15:18:29.202430 6547 tablet_service.cc:1514] Processing DeleteTablet for tablet bcbeb038ca9a45aca857b81d98ebca81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:29 UTC) from {username='oryx', principal='oryx/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:51048
I20260924 15:18:29.202709 6411 tablet_service.cc:1514] Processing DeleteTablet for tablet bcbeb038ca9a45aca857b81d98ebca81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:29 UTC) from {username='oryx', principal='oryx/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:38930
I20260924 15:18:29.202368 6685 tablet_service.cc:1514] Processing DeleteTablet for tablet bcbeb038ca9a45aca857b81d98ebca81 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:18:29 UTC) from {username='oryx', principal='oryx/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:36360
I20260924 15:18:29.203276 6801 tablet_replica.cc:330] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a: stopping tablet replica
I20260924 15:18:29.203385 6801 raft_consensus.cc:2229] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:18:29.203493 6801 raft_consensus.cc:2258] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:29.203729 6799 tablet_replica.cc:330] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06: stopping tablet replica
I20260924 15:18:29.203816 6799 raft_consensus.cc:2229] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:18:29.203903 6799 raft_consensus.cc:2258] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:29.204030 6801 ts_tablet_manager.cc:1905] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:29.204101 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 5466
I20260924 15:18:29.204149 6800 tablet_replica.cc:330] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd: stopping tablet replica
I20260924 15:18:29.205475 6799 ts_tablet_manager.cc:1905] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:29.206981 6800 raft_consensus.cc:2229] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:18:29.207157 6800 raft_consensus.cc:2258] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:18:29.207252 6799 ts_tablet_manager.cc:1918] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:18:29.207379 6799 log.cc:1196] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/bcbeb038ca9a45aca857b81d98ebca81
I20260924 15:18:29.207655 6800 ts_tablet_manager.cc:1905] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:18:29.207815 6799 ts_tablet_manager.cc:1939] T bcbeb038ca9a45aca857b81d98ebca81 P a2dfc048810b439d880bf058dfda0a06: Deleting consensus metadata
I20260924 15:18:29.209244 6800 ts_tablet_manager.cc:1918] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:18:29.209369 6800 log.cc:1196] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/bcbeb038ca9a45aca857b81d98ebca81
I20260924 15:18:29.209739 6800 ts_tablet_manager.cc:1939] T bcbeb038ca9a45aca857b81d98ebca81 P eb0abe6f3cda4468b8e17438d5d184bd: Deleting consensus metadata
W20260924 15:18:29.211369 6501 connection.cc:535] server connection from 127.0.0.1:51048 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 15:18:29.211472 6641 connection.cc:535] server connection from 127.0.0.1:36360 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 15:18:29.215736 6801 ts_tablet_manager.cc:1918] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:18:29.215896 6801 log.cc:1196] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0/minicluster-data/ts-0/wal/wals/bcbeb038ca9a45aca857b81d98ebca81
I20260924 15:18:29.216290 6801 ts_tablet_manager.cc:1939] T bcbeb038ca9a45aca857b81d98ebca81 P 27c5c2f9a521468c87e1cbbdcd730d7a: Deleting consensus metadata
W20260924 15:18:29.216734 6365 connection.cc:441] server connection from 127.0.0.1:38930 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38930 (request call id 1) could send its response
I20260924 15:18:29.217739 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.203391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53530 (local address 127.31.12.124:40647)
0924 15:18:29.203448 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:29.203450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:29.209205 (+ 5755us) server_negotiation.cc:407] Connection header received
0924 15:18:29.209320 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:29.209322 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:29.209334 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:29.209378 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:29.210230 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.210731 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.211450 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.211727 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.211768 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.213420 (+ 1652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:29.213421 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:29.213422 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:29.213429 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:29.215121 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:29.215535 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:29.215536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:29.215536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:29.215596 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:29.215915 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:29.215916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:29.215916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:29.216065 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:29.216141 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:29.217566 (+ 1425us) server_negotiation.cc:299] Negotiation successful
0924 15:18:29.217647 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:18:29.222404 6144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.213821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43126 (local address 127.31.12.125:35423)
0924 15:18:29.213883 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:29.213886 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:29.214262 (+ 376us) server_negotiation.cc:407] Connection header received
0924 15:18:29.214357 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:29.214358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:29.214368 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:29.214395 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:29.215365 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.216089 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.217286 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.217607 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.217664 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.219430 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:29.219432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:29.219432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:29.219439 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:29.221173 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:29.221601 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:29.221603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:29.221603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:29.221659 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:29.221869 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:29.221870 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:29.221871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:29.222010 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:29.222111 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:29.222213 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 15:18:29.222285 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:18:29.223261 6808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:29.213954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53544 (local address 127.31.12.124:40647)
0924 15:18:29.214162 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 15:18:29.214164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:18:29.214349 (+ 185us) server_negotiation.cc:407] Connection header received
0924 15:18:29.214425 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:18:29.214426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:18:29.215796 (+ 1370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:18:29.215855 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:18:29.216900 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.217447 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.218410 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:18:29.218719 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:18:29.218793 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:18:29.220091 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:18:29.220092 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:18:29.220093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:18:29.220099 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:18:29.221949 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:29.222316 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:29.222317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:29.222318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:29.222370 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:18:29.222760 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:18:29.222761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:18:29.222762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:18:29.222876 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:18:29.222975 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 15:18:29.223082 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 15:18:29.223160 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":46,"threads_started":1}
I20260924 15:18:30.199698 6810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:30.199403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:47025)
0924 15:18:30.199558 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:30.199645 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
W20260924 15:18:30.199957 6478 heartbeater.cc:644] Failed to heartbeat to 127.31.12.126:37657 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
I20260924 15:18:30.203173 6811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:30.202944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:39213)
0924 15:18:30.203051 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:30.203136 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":67,"threads_started":1}
W20260924 15:18:30.203351 6617 heartbeater.cc:644] Failed to heartbeat to 127.31.12.126:37657 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
I20260924 15:18:30.203675 6812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:30.203450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:50791)
0924 15:18:30.203565 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:30.203621 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
W20260924 15:18:30.203841 6754 heartbeater.cc:644] Failed to heartbeat to 127.31.12.126:37657 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
I20260924 15:18:30.702378 6813 raft_consensus.cc:482] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b6f5737f73c4374a92ee2c8a69bac93)
I20260924 15:18:30.702561 6813 raft_consensus.cc:504] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: false } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } }
I20260924 15:18:30.702812 6813 leader_election.cc:288] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b6f5737f73c4374a92ee2c8a69bac93 (127.31.12.126:37657)
I20260924 15:18:30.703200 6814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:30.702959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.0.0.1:47390)
0924 15:18:30.703083 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:30.703155 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
W20260924 15:18:30.703341 6093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
I20260924 15:18:30.703739 6814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:30.703637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.0.0.1:47400)
0924 15:18:30.703693 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:30.703720 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260924 15:18:30.703840 6093 leader_election.cc:334] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4b6f5737f73c4374a92ee2c8a69bac93 (127.31.12.126:37657): Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
I20260924 15:18:30.703902 6093 leader_election.cc:302] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: efed9eeea550446e838d24f9dc145216; no voters: 4b6f5737f73c4374a92ee2c8a69bac93
I20260924 15:18:30.704020 6813 raft_consensus.cc:2726] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:18:31.200528 6816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:31.200300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:54841)
0924 15:18:31.200414 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:31.200458 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
I20260924 15:18:31.203855 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:31.203650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:39133)
0924 15:18:31.203771 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:31.203816 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260924 15:18:31.204340 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:31.204171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:59731)
0924 15:18:31.204254 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:31.204306 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68,"thread_start_us":44,"threads_started":1}
I20260924 15:18:32.201303 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:32.201023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:50857)
0924 15:18:32.201211 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:32.201260 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":84,"threads_started":1}
I20260924 15:18:32.204432 6820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:32.204246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:50225)
0924 15:18:32.204357 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:32.204393 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":51,"threads_started":1}
I20260924 15:18:32.204841 6821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:32.204688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:47565)
0924 15:18:32.204775 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:32.204808 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70,"thread_start_us":49,"threads_started":1}
I20260924 15:18:32.347887 6822 raft_consensus.cc:482] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b6f5737f73c4374a92ee2c8a69bac93)
I20260924 15:18:32.347990 6822 raft_consensus.cc:504] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: false } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } }
I20260924 15:18:32.348121 6822 leader_election.cc:288] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b6f5737f73c4374a92ee2c8a69bac93 (127.31.12.126:37657)
I20260924 15:18:32.348484 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:32.348307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.0.0.1:47612)
0924 15:18:32.348411 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:32.348443 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":51,"threads_started":1}
I20260924 15:18:32.348915 6823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:32.348807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.0.0.1:47628)
0924 15:18:32.348860 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:32.348894 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 15:18:32.349017 6093 leader_election.cc:334] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4b6f5737f73c4374a92ee2c8a69bac93 (127.31.12.126:37657): Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
I20260924 15:18:32.349092 6093 leader_election.cc:302] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: efed9eeea550446e838d24f9dc145216; no voters: 4b6f5737f73c4374a92ee2c8a69bac93
I20260924 15:18:32.349181 6822 raft_consensus.cc:2726] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:18:33.201959 6825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:33.201675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:46713)
0924 15:18:33.201812 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:33.201877 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260924 15:18:33.204968 6826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:33.204776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:48621)
0924 15:18:33.204895 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:33.204931 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
I20260924 15:18:33.205363 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:33.205205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:45575)
0924 15:18:33.205298 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:33.205330 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78,"thread_start_us":42,"threads_started":1}
I20260924 15:18:34.202702 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:34.202428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:57377)
0924 15:18:34.202575 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:34.202625 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":70,"threads_started":1}
I20260924 15:18:34.205685 6829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:34.205406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:36443)
0924 15:18:34.205569 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:34.205638 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":96,"threads_started":1}
I20260924 15:18:34.205978 6830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:34.205753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:43217)
0924 15:18:34.205896 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:34.205946 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260924 15:18:35.189256 6831 raft_consensus.cc:482] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b6f5737f73c4374a92ee2c8a69bac93)
I20260924 15:18:35.189363 6831 raft_consensus.cc:504] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: false } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } }
I20260924 15:18:35.189599 6831 leader_election.cc:288] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b6f5737f73c4374a92ee2c8a69bac93 (127.31.12.126:37657)
I20260924 15:18:35.189949 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:35.189788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.0.0.1:47860)
0924 15:18:35.189878 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:35.189909 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":38,"threads_started":1}
I20260924 15:18:35.190433 6832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:35.190333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.0.0.1:47862)
0924 15:18:35.190386 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:35.190414 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 15:18:35.190575 6093 leader_election.cc:334] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4b6f5737f73c4374a92ee2c8a69bac93 (127.31.12.126:37657): Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
I20260924 15:18:35.190635 6093 leader_election.cc:302] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: efed9eeea550446e838d24f9dc145216; no voters: 4b6f5737f73c4374a92ee2c8a69bac93
I20260924 15:18:35.190732 6831 raft_consensus.cc:2726] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:18:35.203446 6834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:35.203218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:49025)
0924 15:18:35.203354 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:35.203391 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":78,"threads_started":1}
I20260924 15:18:35.206308 6835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:35.206119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:45515)
0924 15:18:35.206246 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:35.206275 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260924 15:18:35.206550 6836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:35.206365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:39187)
0924 15:18:35.206474 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:35.206509 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
I20260924 15:18:36.204180 6837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:36.203912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:50063)
0924 15:18:36.204083 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:36.204130 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":112,"threads_started":1}
I20260924 15:18:36.206805 6838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:36.206632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:52931)
0924 15:18:36.206737 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:36.206772 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":49,"threads_started":1}
I20260924 15:18:36.207132 6839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:36.206973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:59551)
0924 15:18:36.207070 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:36.207101 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":51,"threads_started":1}
I20260924 15:18:37.205116 6841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:37.204810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:34487)
0924 15:18:37.204976 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:37.205053 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
I20260924 15:18:37.207439 6842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:37.207223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:44157)
0924 15:18:37.207344 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:37.207388 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":51,"threads_started":1}
I20260924 15:18:37.207693 6843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:37.207474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:41869)
0924 15:18:37.207616 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:37.207647 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260924 15:18:38.206278 6844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:38.205807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:58963)
0924 15:18:38.205975 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:38.206033 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
I20260924 15:18:38.207988 6845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:38.207770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:60927)
0924 15:18:38.207876 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:38.207928 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":47,"threads_started":1}
I20260924 15:18:38.208247 6846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:38.208074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:41891)
0924 15:18:38.208184 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:38.208215 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":62,"threads_started":1}
I20260924 15:18:39.207486 6847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:39.207269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:46877)
0924 15:18:39.207385 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:39.207428 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":66,"threads_started":1}
I20260924 15:18:39.208562 6848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:39.208351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:54945)
0924 15:18:39.208465 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:39.208513 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":57,"threads_started":1}
I20260924 15:18:39.208815 6849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:39.208599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:59193)
0924 15:18:39.208733 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:39.208770 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260924 15:18:40.127744 6850 raft_consensus.cc:482] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b6f5737f73c4374a92ee2c8a69bac93)
I20260924 15:18:40.127851 6850 raft_consensus.cc:504] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: false } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } }
I20260924 15:18:40.128018 6850 leader_election.cc:288] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b6f5737f73c4374a92ee2c8a69bac93 (127.31.12.126:37657)
I20260924 15:18:40.128317 6851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:40.128144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.0.0.1:58608)
0924 15:18:40.128253 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:40.128282 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":65,"threads_started":1}
W20260924 15:18:40.128400 6093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111) [suppressed 2 similar messages]
I20260924 15:18:40.128751 6851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:40.128653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.0.0.1:58614)
0924 15:18:40.128701 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:40.128733 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 15:18:40.128826 6093 leader_election.cc:334] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4b6f5737f73c4374a92ee2c8a69bac93 (127.31.12.126:37657): Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
I20260924 15:18:40.128902 6093 leader_election.cc:302] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: efed9eeea550446e838d24f9dc145216; no voters: 4b6f5737f73c4374a92ee2c8a69bac93
I20260924 15:18:40.129001 6850 raft_consensus.cc:2726] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:18:40.208372 6853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:40.208103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:54041)
0924 15:18:40.208284 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:40.208330 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":117,"threads_started":1}
I20260924 15:18:40.209153 6854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:40.208960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:47199)
0924 15:18:40.209087 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:40.209119 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":49,"threads_started":1}
I20260924 15:18:40.209352 6855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:40.209146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:53219)
0924 15:18:40.209271 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:40.209302 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260924 15:18:41.209034 6856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:41.208805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:42781)
0924 15:18:41.208954 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:41.208991 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
I20260924 15:18:41.209689 6857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:41.209508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:60489)
0924 15:18:41.209620 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:41.209654 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260924 15:18:41.210027 6858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:41.209771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:55967)
0924 15:18:41.209955 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:41.209994 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":63,"threads_started":1}
I20260924 15:18:42.209668 6859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:42.209427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:47785)
0924 15:18:42.209573 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:42.209624 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260924 15:18:42.210287 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:42.210111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:43011)
0924 15:18:42.210215 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:42.210250 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":47,"threads_started":1}
I20260924 15:18:42.210629 6861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:42.210462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:33661)
0924 15:18:42.210575 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:42.210599 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260924 15:18:43.210384 6862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:43.210158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:53793)
0924 15:18:43.210300 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:43.210341 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
I20260924 15:18:43.210960 6863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:43.210719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:37005)
0924 15:18:43.210854 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:43.210913 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
I20260924 15:18:43.211202 6864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:43.211013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:54887)
0924 15:18:43.211126 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:43.211161 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":68,"threads_started":1}
I20260924 15:18:44.211268 6865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:44.210977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:45201)
0924 15:18:44.211139 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:44.211212 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":69,"threads_started":1}
I20260924 15:18:44.211675 6866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:44.211467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:55467)
0924 15:18:44.211576 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:44.211629 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":60,"threads_started":1}
I20260924 15:18:44.211747 6867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:44.211569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:38233)
0924 15:18:44.211661 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:44.211694 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":52,"threads_started":1}
I20260924 15:18:45.211975 6868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:45.211693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:42637)
0924 15:18:45.211846 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:45.211916 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20260924 15:18:45.212265 6870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:45.212076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:33495)
0924 15:18:45.212189 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:45.212231 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":48,"threads_started":1}
I20260924 15:18:45.212319 6869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:45.212062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:46925)
0924 15:18:45.212195 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:45.212233 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":51,"threads_started":1}
I20260924 15:18:46.005810 6871 raft_consensus.cc:482] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b6f5737f73c4374a92ee2c8a69bac93)
I20260924 15:18:46.005926 6871 raft_consensus.cc:504] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: false } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } }
I20260924 15:18:46.006075 6871 leader_election.cc:288] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b6f5737f73c4374a92ee2c8a69bac93 (127.31.12.126:37657)
I20260924 15:18:46.006400 6872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:46.006219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.0.0.1:58900)
0924 15:18:46.006319 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:46.006362 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":47,"threads_started":1}
W20260924 15:18:46.006495 6093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
I20260924 15:18:46.006898 6872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:46.006801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.0.0.1:58910)
0924 15:18:46.006850 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:46.006878 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 15:18:46.006980 6093 leader_election.cc:334] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4b6f5737f73c4374a92ee2c8a69bac93 (127.31.12.126:37657): Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
I20260924 15:18:46.007067 6093 leader_election.cc:302] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: efed9eeea550446e838d24f9dc145216; no voters: 4b6f5737f73c4374a92ee2c8a69bac93
I20260924 15:18:46.007145 6871 raft_consensus.cc:2726] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:18:46.212735 6874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:46.212488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:45017)
0924 15:18:46.212630 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:46.212676 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260924 15:18:46.212855 6876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:46.212667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:51465)
0924 15:18:46.212788 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:46.212815 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":47,"threads_started":1}
I20260924 15:18:46.212926 6875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:46.212648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:47439)
0924 15:18:46.212765 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:46.212807 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":52,"threads_started":1}
I20260924 15:18:47.213403 6877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:47.213119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:41615)
0924 15:18:47.213297 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:47.213342 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":108,"threads_started":1}
I20260924 15:18:47.213496 6878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:47.213300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:33927)
0924 15:18:47.213392 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:47.213429 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":47,"threads_started":1}
I20260924 15:18:47.213511 6879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:47.213299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:49371)
0924 15:18:47.213414 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:47.213466 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":53,"threads_started":1}
I20260924 15:18:48.214114 6880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:48.213873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:38697)
0924 15:18:48.214022 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:48.214062 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":62,"threads_started":1}
I20260924 15:18:48.214099 6881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:48.213868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:34251)
0924 15:18:48.214012 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:48.214057 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260924 15:18:48.214234 6882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:48.214030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:36603)
0924 15:18:48.214161 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:48.214191 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":45,"threads_started":1}
I20260924 15:18:49.214854 6884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:49.214597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:60151)
0924 15:18:49.214751 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:49.214797 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260924 15:18:49.214848 6883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:49.214596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:55761)
0924 15:18:49.214751 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:49.214796 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260924 15:18:49.214848 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:49.214604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:51065)
0924 15:18:49.214770 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:49.214801 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":72,"threads_started":1}
I20260924 15:18:50.215503 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:50.215249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:45619)
0924 15:18:50.215395 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:50.215441 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260924 15:18:50.215516 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:50.215304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:44333)
0924 15:18:50.215410 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:50.215467 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":47,"threads_started":1}
I20260924 15:18:50.215550 6888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:50.215361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:38581)
0924 15:18:50.215474 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:50.215507 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
I20260924 15:18:51.216218 6889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:51.215940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:60673)
0924 15:18:51.216116 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:51.216159 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":64,"threads_started":1}
I20260924 15:18:51.216277 6891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:51.216038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:34555)
0924 15:18:51.216206 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:51.216236 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":57,"threads_started":1}
I20260924 15:18:51.216390 6890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:51.215965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:41801)
0924 15:18:51.216303 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:51.216341 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":62,"threads_started":1}
I20260924 15:18:52.216949 6892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:52.216707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:39793)
0924 15:18:52.216839 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:52.216882 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260924 15:18:52.217154 6894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:52.216866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:45435)
0924 15:18:52.217068 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:52.217104 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":60,"threads_started":1}
I20260924 15:18:52.217217 6893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:52.216866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:45479)
0924 15:18:52.217145 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:52.217175 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":52,"threads_started":1}
I20260924 15:18:53.217994 6895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:53.217588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:57937)
0924 15:18:53.217866 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:53.217929 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":64,"threads_started":1}
I20260924 15:18:53.218147 6896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:53.217614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:48035)
0924 15:18:53.218057 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:53.218097 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":61,"threads_started":1}
I20260924 15:18:53.218214 6897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:53.217846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:60661)
0924 15:18:53.218138 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:53.218170 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":49,"threads_started":1}
I20260924 15:18:53.263821 6898 raft_consensus.cc:482] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b6f5737f73c4374a92ee2c8a69bac93)
I20260924 15:18:53.263985 6898 raft_consensus.cc:504] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b6f5737f73c4374a92ee2c8a69bac93" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37657 } } peers { permanent_uuid: "efed9eeea550446e838d24f9dc145216" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 35423 } attrs { promote: false } } peers { permanent_uuid: "0cc1c8105976413c9be36d2d9f807985" member_type: NON_VOTER last_known_addr { host: "127.31.12.124" port: 40647 } attrs { promote: true } }
I20260924 15:18:53.264213 6898 leader_election.cc:288] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b6f5737f73c4374a92ee2c8a69bac93 (127.31.12.126:37657)
I20260924 15:18:53.264597 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:53.264406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.0.0.1:54818)
0924 15:18:53.264513 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:53.264559 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":59,"threads_started":1}
W20260924 15:18:53.264686 6093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
I20260924 15:18:53.265056 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:53.264943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.0.0.1:54832)
0924 15:18:53.265002 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:53.265028 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260924 15:18:53.265136 6093 leader_election.cc:334] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4b6f5737f73c4374a92ee2c8a69bac93 (127.31.12.126:37657): Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
I20260924 15:18:53.265179 6093 leader_election.cc:302] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: efed9eeea550446e838d24f9dc145216; no voters: 4b6f5737f73c4374a92ee2c8a69bac93
I20260924 15:18:53.265235 6898 raft_consensus.cc:2726] T 00000000000000000000000000000000 P efed9eeea550446e838d24f9dc145216 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:18:54.219000 6902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:54.218579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:51615)
0924 15:18:54.218905 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:54.218949 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":43,"threads_started":1}
I20260924 15:18:54.219012 6903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:54.218730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:37671)
0924 15:18:54.218923 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:54.218950 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":58,"threads_started":1}
I20260924 15:18:54.219064 6901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:54.218530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:35955)
0924 15:18:54.218972 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:54.219014 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":62,"threads_started":1}
I20260924 15:18:55.219974 6904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:55.219607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:43841)
0924 15:18:55.219857 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:55.219898 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":65,"threads_started":1}
I20260924 15:18:55.219969 6906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:55.219725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:36581)
0924 15:18:55.219866 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:55.219911 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":43,"threads_started":1}
I20260924 15:18:55.220027 6905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:55.219581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:41251)
0924 15:18:55.219947 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:55.219985 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":98,"threads_started":1}
I20260924 15:18:56.220857 6908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:56.220445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:52073)
0924 15:18:56.220747 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:56.220794 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":76,"threads_started":1}
I20260924 15:18:56.220860 6907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:56.220410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:41127)
0924 15:18:56.220765 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:56.220807 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":62,"threads_started":1}
I20260924 15:18:56.221026 6909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:56.220677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:43003)
0924 15:18:56.220905 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:56.220941 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":68,"threads_started":1}
I20260924 15:18:57.221700 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:57.221464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:53049)
0924 15:18:57.221586 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:57.221641 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260924 15:18:57.221753 6912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:57.221539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:38845)
0924 15:18:57.221661 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:57.221700 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
I20260924 15:18:57.221752 6911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:57.221539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:52531)
0924 15:18:57.221677 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:57.221712 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":56,"threads_started":1}
I20260924 15:18:58.222376 6913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:58.222126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:54289)
0924 15:18:58.222268 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:58.222313 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
I20260924 15:18:58.222667 6915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:58.222391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:52823)
0924 15:18:58.222566 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:58.222612 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":61,"threads_started":1}
I20260924 15:18:58.222692 6914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:58.222189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:56041)
0924 15:18:58.222615 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:58.222653 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":57,"threads_started":1}
I20260924 15:18:59.223227 6916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:59.222910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.66:41199)
0924 15:18:59.223077 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:59.223120 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":69,"threads_started":1}
I20260924 15:18:59.223589 6917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:59.223337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.65:51287)
0924 15:18:59.223505 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:59.223542 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":60,"threads_started":1}
I20260924 15:18:59.223829 6918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:18:59.223621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37657 (local address 127.31.12.67:38301)
0924 15:18:59.223748 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 15:18:59.223786 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:37657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:903: Failure
Failed
Bad status: Timed out: Could not connect to the cluster: ConnectToClusterRpc(addrs: 127.31.12.126:37657,127.31.12.125:35423,127.31.12.124:40647, num_attempts: 243) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.31.12.126:37657,127.31.12.125:35423,127.31.12.124:40647, num_attempts: 1)
I20260924 15:18:59.414224 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 6258
I20260924 15:18:59.419592 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 6485
I20260924 15:18:59.424981 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 6623
I20260924 15:18:59.429589 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 5967
I20260924 15:18:59.434993 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 6162
2026-09-24T15:18:59Z chronyd exiting
I20260924 15:18:59.451437 31793 test_util.cc:175] -----------------------------------------------
I20260924 15:18:59.451508 31793 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790263080303576-31793-0
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (33047 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:18:59 dist-test-slave-zwbb krb5kdc[6921](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:18:59 dist-test-slave-zwbb krb5kdc[6921](info): set up 2 sockets
Sep 24 15:18:59 dist-test-slave-zwbb krb5kdc[6921](info): commencing operation
krb5kdc: starting...
W20260924 15:19:00.475762 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:19:00 dist-test-slave-zwbb krb5kdc[6921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:19:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:19:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.126.keytab.
Entry for principal oryx/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.126.keytab.
I20260924 15:19:00.592175 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:43239
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.126.keytab
--principal=oryx/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:42865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:43239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:19:00.672320 6934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:00.672569 6934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:00.672608 6934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:00.673791 6934 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:19:00.673856 6934 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 15:19:00.673877 6934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:00.673900 6934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:00.673923 6934 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:19:00.673996 6934 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:19:00.675534 6934 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:42865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:43239
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.126.keytab
--principal=oryx/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:43239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:00.675724 6934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:00.675884 6934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:00.680980 6942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:00.681022 6940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:00.681010 6939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:00.681061 6934 server_base.cc:1034] running on GCE node
I20260924 15:19:00.681424 6934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:00.681775 6934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:00.682914 6934 hybrid_clock.cc:648] HybridClock initialized: now 1790263140682900 us; error 30 us; skew 500 ppm
Sep 24 15:19:00 dist-test-slave-zwbb krb5kdc[6921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263140, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:19:00.688702 6934 init.cc:377] Logged in from keytab as oryx/127.31.12.126@KRBTEST.COM (short username oryx)
I20260924 15:19:00.689172 6934 webserver.cc:460] Webserver started at http://127.31.12.126:43929/ using document root <none> and password file <none>
I20260924 15:19:00.689391 6934 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:00.689472 6934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:00.689644 6934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:19:00.690706 6934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "e6d9849a34d749fb9d585de3ff9f7974"
format_stamp: "Formatted at 2026-09-24 15:19:00 on dist-test-slave-zwbb"
server_key: "e0a755f73d630dd71f96e851477b3299"
server_key_iv: "c8de339b3aea8ebd67f815eb1186f4e3"
server_key_version: "encryptionkey@0"
I20260924 15:19:00.691073 6934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "e6d9849a34d749fb9d585de3ff9f7974"
format_stamp: "Formatted at 2026-09-24 15:19:00 on dist-test-slave-zwbb"
server_key: "e0a755f73d630dd71f96e851477b3299"
server_key_iv: "c8de339b3aea8ebd67f815eb1186f4e3"
server_key_version: "encryptionkey@0"
I20260924 15:19:00.692404 6934 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:19:00.693141 6949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:00.693318 6934 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:19:00.693434 6934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "e6d9849a34d749fb9d585de3ff9f7974"
format_stamp: "Formatted at 2026-09-24 15:19:00 on dist-test-slave-zwbb"
server_key: "e0a755f73d630dd71f96e851477b3299"
server_key_iv: "c8de339b3aea8ebd67f815eb1186f4e3"
server_key_version: "encryptionkey@0"
I20260924 15:19:00.693564 6934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:00.734174 6934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:00.736483 6934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:00.736651 6934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:00.741262 6934 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:43239
I20260924 15:19:00.741312 7012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:43239 every 8 connection(s)
I20260924 15:19:00.741644 6934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:19:00.742257 7013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:00.744691 7013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974: Bootstrap starting.
I20260924 15:19:00.745301 7013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:00.745656 7013 log.cc:824] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974: Log is configured to *not* fsync() on all Append() calls
I20260924 15:19:00.746311 7013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974: No bootstrap required, opened a new log
I20260924 15:19:00.747116 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 6934
I20260924 15:19:00.747220 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:19:00.747413 31793 external_mini_cluster.cc:1428] Setting key ca8d7fdd174927fd35bcc27b6d5118b3
I20260924 15:19:00.748889 7013 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d9849a34d749fb9d585de3ff9f7974" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43239 } }
I20260924 15:19:00.749285 7013 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:00.749322 7013 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6d9849a34d749fb9d585de3ff9f7974, State: Initialized, Role: FOLLOWER
I20260924 15:19:00.749498 7013 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d9849a34d749fb9d585de3ff9f7974" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43239 } }
I20260924 15:19:00.749577 7013 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:19:00.749601 7013 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:19:00.749667 7013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:00.750236 7013 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d9849a34d749fb9d585de3ff9f7974" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43239 } }
I20260924 15:19:00.750322 7013 leader_election.cc:302] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6d9849a34d749fb9d585de3ff9f7974; no voters:
I20260924 15:19:00.750779 7013 leader_election.cc:288] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:19:00.750808 7018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:19:00.750933 7018 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974 [term 1 LEADER]: Becoming Leader. State: Replica: e6d9849a34d749fb9d585de3ff9f7974, State: Running, Role: LEADER
I20260924 15:19:00.751077 7013 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:19:00.751099 7018 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d9849a34d749fb9d585de3ff9f7974" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43239 } }
I20260924 15:19:00.751552 7019 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6d9849a34d749fb9d585de3ff9f7974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d9849a34d749fb9d585de3ff9f7974" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43239 } } }
I20260924 15:19:00.751729 7019 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974 [sys.catalog]: This master's current role is: LEADER
I20260924 15:19:00.751585 7020 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6d9849a34d749fb9d585de3ff9f7974. Latest consensus state: current_term: 1 leader_uuid: "e6d9849a34d749fb9d585de3ff9f7974" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6d9849a34d749fb9d585de3ff9f7974" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43239 } } }
I20260924 15:19:00.751994 7020 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974 [sys.catalog]: This master's current role is: LEADER
I20260924 15:19:00.752000 7027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:19:00.752646 7027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:19:00.754331 7027 catalog_manager.cc:1347] Generated new cluster ID: cc6ad16d2b034551aa4cc54a2e48c8ff
I20260924 15:19:00.754386 7027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:19:00 dist-test-slave-zwbb krb5kdc[6921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263140, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM
I20260924 15:19:00.758854 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:00.747916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35006 (local address 127.31.12.126:43239)
0924 15:19:00.748128 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:00.748137 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:00.748169 (+ 32us) server_negotiation.cc:407] Connection header received
0924 15:19:00.748869 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:00.748875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:00.748921 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:00.749084 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:00.750024 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:00.751114 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:00.752341 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:00.752983 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:00.753119 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:00.755648 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:00.755652 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:00.755653 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:19:00.755664 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:00.757295 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:00.757743 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:00.757746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:00.757747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:00.757845 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:00.758056 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:00.758057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:00.758057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:00.758282 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:00.758469 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:00.758547 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 15:19:00.758697 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
I20260924 15:19:00.760514 7027 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:19:00.760994 7027 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:19:00.768251 7027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974: Generated new TSK 0
I20260924 15:19:00.768415 7027 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.65.keytab.
Entry for principal oryx/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.65.keytab.
I20260924 15:19:00.798307 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:43239
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.65.keytab
--principal=oryx/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:42865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:19:00.897380 7039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:00.897646 7039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:00.897686 7039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:00.898815 7039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:00.898876 7039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:00.898912 7039 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:19:00.898950 7039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:19:00.900956 7039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:42865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.65.keytab
--principal=oryx/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.12.126:43239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:00.901247 7039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:00.901499 7039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:00.907706 7044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:00.907713 7045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:00.907718 7047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:00.910410 7039 server_base.cc:1034] running on GCE node
I20260924 15:19:00.910611 7039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:00.910872 7039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:00.912030 7039 hybrid_clock.cc:648] HybridClock initialized: now 1790263140912015 us; error 23 us; skew 500 ppm
Sep 24 15:19:00 dist-test-slave-zwbb krb5kdc[6921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263140, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:19:00.914440 7039 init.cc:377] Logged in from keytab as oryx/127.31.12.65@KRBTEST.COM (short username oryx)
I20260924 15:19:00.914849 7039 webserver.cc:460] Webserver started at http://127.31.12.65:35489/ using document root <none> and password file <none>
I20260924 15:19:00.915087 7039 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:00.915154 7039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:00.915306 7039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:19:00.916345 7039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "0510948451454cf994e41c6f5390beb0"
format_stamp: "Formatted at 2026-09-24 15:19:00 on dist-test-slave-zwbb"
server_key: "e909976969faa489d6846511684d119b"
server_key_iv: "a8dd98a01765a9a37b58bf7baabf0f86"
server_key_version: "encryptionkey@0"
I20260924 15:19:00.916754 7039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "0510948451454cf994e41c6f5390beb0"
format_stamp: "Formatted at 2026-09-24 15:19:00 on dist-test-slave-zwbb"
server_key: "e909976969faa489d6846511684d119b"
server_key_iv: "a8dd98a01765a9a37b58bf7baabf0f86"
server_key_version: "encryptionkey@0"
I20260924 15:19:00.918154 7039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:19:00.918988 7054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:00.919176 7039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:19:00.919279 7039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "0510948451454cf994e41c6f5390beb0"
format_stamp: "Formatted at 2026-09-24 15:19:00 on dist-test-slave-zwbb"
server_key: "e909976969faa489d6846511684d119b"
server_key_iv: "a8dd98a01765a9a37b58bf7baabf0f86"
server_key_version: "encryptionkey@0"
I20260924 15:19:00.919370 7039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:00.928529 7039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:00.930411 7039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:00.930562 7039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:00.931056 7039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:19:00.931116 7039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:00.931149 7039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:19:00.931188 7039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:19:00 dist-test-slave-zwbb krb5kdc[6921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263140, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM
I20260924 15:19:00.938476 7039 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:34463
I20260924 15:19:00.938556 7170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:34463 every 8 connection(s)
I20260924 15:19:00.938922 7039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:19:00.941021 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:00.931466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:48149 (local address 127.31.12.126:43239)
0924 15:19:00.931543 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:00.931546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:00.931703 (+ 157us) server_negotiation.cc:407] Connection header received
0924 15:19:00.932181 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:00.932182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:00.932198 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:00.932277 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:00.933009 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:00.933872 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:00.934909 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:00.935225 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:00.935309 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:00.938544 (+ 3235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:00.938546 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:00.938546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:19:00.938557 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:00.939946 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:00.940297 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:00.940298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:00.940298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:00.940344 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:00.940533 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:00.940534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:00.940534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:00.940633 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:00.940682 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:00.940896 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 15:19:00.940960 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 15:19:00.941242 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:00.931441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43239 (local address 127.31.12.65:48149)
0924 15:19:00.931573 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:00.931611 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:00.932085 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:00.932335 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:00.932338 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:00.932612 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:00.932901 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:00.932907 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:00.933973 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:00.933974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:00.934809 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:00.934812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:00.935321 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:00.935322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:00.935457 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:00.935953 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:00.935962 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:00.938440 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:19:00.940022 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:00.940026 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:00.940028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:00.940234 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:00.940402 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:00.940402 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:00.940402 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:00.940485 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:00.940685 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:19:00.940690 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:19:00.940798 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 15:19:00.940877 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 15:19:00.941002 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
I20260924 15:19:00.941747 7171 heartbeater.cc:347] Connected to a master server at 127.31.12.126:43239
I20260924 15:19:00.941831 7171 heartbeater.cc:464] Registering TS with master...
I20260924 15:19:00.942009 7171 heartbeater.cc:511] Master 127.31.12.126:43239 requested a full tablet report, sending...
I20260924 15:19:00.942601 6965 ts_manager.cc:194] Registered new tserver with Master: 0510948451454cf994e41c6f5390beb0 (127.31.12.65:34463)
I20260924 15:19:00.943425 6965 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:48149
I20260924 15:19:00.943732 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 7039
I20260924 15:19:00.943852 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:19:00.944063 31793 external_mini_cluster.cc:1428] Setting key c323bd4343d08ea3fcae4f3b42673bb1
I20260924 15:19:00.948843 7177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:00.942757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43239 (local address 127.31.12.65:33383)
0924 15:19:00.942869 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:00.942880 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:00.942935 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:00.943060 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:00.943060 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:00.943203 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:00.943250 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:00.943251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:00.944013 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:00.944014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:00.944837 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:00.944840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:00.945178 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:00.945179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:00.945252 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:00.945724 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:00.945730 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:00.946140 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:19:00.947958 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:00.947958 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:00.947959 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:00.948125 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:00.948319 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:00.948319 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:00.948320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:00.948369 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:00.948632 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:19:00.948632 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:19:00.948664 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 15:19:00.948711 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 15:19:00.948778 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":68,"threads_started":1}
I20260924 15:19:00.948967 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:00.942754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:33383 (local address 127.31.12.126:43239)
0924 15:19:00.942794 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:00.942796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:00.942944 (+ 148us) server_negotiation.cc:407] Connection header received
0924 15:19:00.942984 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:00.942985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:00.942994 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:00.943050 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:00.943306 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:00.943915 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:00.944948 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:00.945122 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:00.945176 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:00.946235 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:00.946237 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:00.946237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:19:00.946243 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:00.947869 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:00.948193 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:00.948194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:00.948195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:00.948255 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:00.948426 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:00.948427 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:00.948427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:00.948567 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:00.948630 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:00.948737 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 15:19:00.948823 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
WARNING: no policy specified for oryx/127.31.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.66.keytab.
Entry for principal oryx/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.66.keytab.
I20260924 15:19:00.968210 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:43239
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.66.keytab
--principal=oryx/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:42865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:19:01.050139 7180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:01.050334 7180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:01.050357 7180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:01.051595 7180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:01.051636 7180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:01.051657 7180 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:19:01.051697 7180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:19:01.053345 7180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:42865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.66.keytab
--principal=oryx/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.12.126:43239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:01.053622 7180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:01.053851 7180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:01.058691 7185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:01.058704 7188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:01.058724 7186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:01.058724 7180 server_base.cc:1034] running on GCE node
I20260924 15:19:01.059156 7180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:01.059402 7180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:01.060537 7180 hybrid_clock.cc:648] HybridClock initialized: now 1790263141060506 us; error 50 us; skew 500 ppm
Sep 24 15:19:01 dist-test-slave-zwbb krb5kdc[6921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263141, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:19:01.063126 7180 init.cc:377] Logged in from keytab as oryx/127.31.12.66@KRBTEST.COM (short username oryx)
I20260924 15:19:01.063561 7180 webserver.cc:460] Webserver started at http://127.31.12.66:44631/ using document root <none> and password file <none>
I20260924 15:19:01.063807 7180 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:01.063871 7180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:01.064019 7180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:19:01.065029 7180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "82056c4aa10a4f63b50c264c3e62a0a4"
format_stamp: "Formatted at 2026-09-24 15:19:01 on dist-test-slave-zwbb"
server_key: "1e66487b75d25d391146fb73e257b231"
server_key_iv: "8f8cf4e676b53a77d73ab2a5b84d9a7b"
server_key_version: "encryptionkey@0"
I20260924 15:19:01.065373 7180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "82056c4aa10a4f63b50c264c3e62a0a4"
format_stamp: "Formatted at 2026-09-24 15:19:01 on dist-test-slave-zwbb"
server_key: "1e66487b75d25d391146fb73e257b231"
server_key_iv: "8f8cf4e676b53a77d73ab2a5b84d9a7b"
server_key_version: "encryptionkey@0"
I20260924 15:19:01.066677 7180 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:19:01.067379 7195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:01.067541 7180 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:19:01.067631 7180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "82056c4aa10a4f63b50c264c3e62a0a4"
format_stamp: "Formatted at 2026-09-24 15:19:01 on dist-test-slave-zwbb"
server_key: "1e66487b75d25d391146fb73e257b231"
server_key_iv: "8f8cf4e676b53a77d73ab2a5b84d9a7b"
server_key_version: "encryptionkey@0"
I20260924 15:19:01.067708 7180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:01.084054 7180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:01.086315 7180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:01.086468 7180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:01.086966 7180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:19:01.087030 7180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:01.087073 7180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:19:01.087103 7180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 15:19:01 dist-test-slave-zwbb krb5kdc[6921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263141, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM
I20260924 15:19:01.093705 7180 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:35055
I20260924 15:19:01.093806 7310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:35055 every 8 connection(s)
I20260924 15:19:01.094136 7180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:19:01.096661 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.087466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:35237 (local address 127.31.12.126:43239)
0924 15:19:01.087513 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:01.087515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:01.087754 (+ 239us) server_negotiation.cc:407] Connection header received
0924 15:19:01.088275 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:01.088277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:01.088291 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:01.088341 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:01.089183 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.089863 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.091436 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.091765 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.091851 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.094094 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:01.094095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:01.094096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:19:01.094104 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:01.095506 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:01.095892 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:01.095893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:01.095893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:01.095945 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:01.096116 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:01.096117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:01.096117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:01.096216 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:01.096295 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:01.096531 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 15:19:01.096600 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:19:01.096829 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.087463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43239 (local address 127.31.12.66:35237)
0924 15:19:01.087632 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:01.087672 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:01.088162 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:01.088436 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:01.088440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:01.088802 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:01.089082 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.089090 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.089955 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.089956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.091320 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.091322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.091860 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.091861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.091962 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.092451 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:01.092460 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:01.094009 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:19:01.095606 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:01.095613 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:01.095615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:01.095827 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:01.095987 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:01.095987 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:01.095988 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:01.096069 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:01.096290 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:19:01.096295 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:19:01.096413 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 15:19:01.096495 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:19:01.096608 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260924 15:19:01.097312 7311 heartbeater.cc:347] Connected to a master server at 127.31.12.126:43239
I20260924 15:19:01.097397 7311 heartbeater.cc:464] Registering TS with master...
I20260924 15:19:01.097600 7311 heartbeater.cc:511] Master 127.31.12.126:43239 requested a full tablet report, sending...
I20260924 15:19:01.097976 6966 ts_manager.cc:194] Registered new tserver with Master: 82056c4aa10a4f63b50c264c3e62a0a4 (127.31.12.66:35055)
I20260924 15:19:01.098464 6966 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:35237
I20260924 15:19:01.102739 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 7180
I20260924 15:19:01.102854 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:19:01.103078 31793 external_mini_cluster.cc:1428] Setting key 344c62515ff877133b6cd159c87d981b
I20260924 15:19:01.103295 7317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.098401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43239 (local address 127.31.12.66:41497)
0924 15:19:01.098520 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:01.098531 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:01.098592 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:01.098711 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:01.098711 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:01.098926 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:01.098998 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.099000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.099605 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.099606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.100083 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.100084 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.100362 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.100362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.100405 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.100715 (+ 310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:01.100718 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:01.100976 (+ 258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:19:01.102200 (+ 1224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:01.102201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:01.102201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:01.102378 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:01.102532 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:01.102533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:01.102533 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:01.102577 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:01.102937 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:19:01.102939 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:19:01.103009 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 15:19:01.103082 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 15:19:01.103189 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":65,"threads_started":1}
I20260924 15:19:01.103335 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.098464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:41497 (local address 127.31.12.126:43239)
0924 15:19:01.098508 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:01.098510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:01.098581 (+ 71us) server_negotiation.cc:407] Connection header received
0924 15:19:01.098640 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:01.098641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:01.098651 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:01.098709 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:01.099072 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.099512 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.100145 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.100305 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.100348 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.101040 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:01.101041 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:01.101041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:19:01.101046 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:01.102138 (+ 1092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:01.102437 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:01.102438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:01.102438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:01.102483 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:01.102636 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:01.102637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:01.102637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:01.102829 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:01.102903 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:01.103133 (+ 230us) server_negotiation.cc:299] Negotiation successful
0924 15:19:01.103214 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
WARNING: no policy specified for oryx/127.31.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.67.keytab.
Entry for principal oryx/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.67.keytab.
I20260924 15:19:01.128723 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:43239
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.67.keytab
--principal=oryx/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:42865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:19:01.208977 7320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:01.209203 7320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:01.209244 7320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:01.210457 7320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:01.210515 7320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:01.210551 7320 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 15:19:01.210587 7320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:19:01.212040 7320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:42865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.67.keytab
--principal=oryx/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.31.12.126:43239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:01.212285 7320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:01.212507 7320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:01.217350 7325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:01.217317 7326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:01.217317 7320 server_base.cc:1034] running on GCE node
W20260924 15:19:01.217340 7328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:01.217772 7320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:01.217970 7320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:01.219098 7320 hybrid_clock.cc:648] HybridClock initialized: now 1790263141219076 us; error 39 us; skew 500 ppm
Sep 24 15:19:01 dist-test-slave-zwbb krb5kdc[6921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263141, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:19:01.221576 7320 init.cc:377] Logged in from keytab as oryx/127.31.12.67@KRBTEST.COM (short username oryx)
I20260924 15:19:01.221995 7320 webserver.cc:460] Webserver started at http://127.31.12.67:41187/ using document root <none> and password file <none>
I20260924 15:19:01.222223 7320 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:01.222268 7320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:01.222453 7320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:19:01.223543 7320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "c97581888e264d1fa8711d8c8b66736a"
format_stamp: "Formatted at 2026-09-24 15:19:01 on dist-test-slave-zwbb"
server_key: "7a1a74505b0557595abf0904d4549cf8"
server_key_iv: "720a5754cf2041314ac7feac41d149b6"
server_key_version: "encryptionkey@0"
I20260924 15:19:01.223893 7320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "c97581888e264d1fa8711d8c8b66736a"
format_stamp: "Formatted at 2026-09-24 15:19:01 on dist-test-slave-zwbb"
server_key: "7a1a74505b0557595abf0904d4549cf8"
server_key_iv: "720a5754cf2041314ac7feac41d149b6"
server_key_version: "encryptionkey@0"
I20260924 15:19:01.225198 7320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:19:01.225961 7335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:01.226138 7320 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:01.226226 7320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "c97581888e264d1fa8711d8c8b66736a"
format_stamp: "Formatted at 2026-09-24 15:19:01 on dist-test-slave-zwbb"
server_key: "7a1a74505b0557595abf0904d4549cf8"
server_key_iv: "720a5754cf2041314ac7feac41d149b6"
server_key_version: "encryptionkey@0"
I20260924 15:19:01.226291 7320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:01.235879 7320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:01.237607 7320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:01.237748 7320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:01.238363 7320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:19:01.238433 7320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:01.238461 7320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:19:01.238487 7320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:01.245659 7320 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:37441
I20260924 15:19:01.245800 7452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:37441 every 8 connection(s)
Sep 24 15:19:01 dist-test-slave-zwbb krb5kdc[6921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263141, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM
I20260924 15:19:01.246048 7320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:19:01.248766 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.238647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:43331 (local address 127.31.12.126:43239)
0924 15:19:01.238697 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:01.238699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:01.238941 (+ 242us) server_negotiation.cc:407] Connection header received
0924 15:19:01.239376 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:01.239377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:01.239391 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:01.239477 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:01.240288 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.241163 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.242750 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.243151 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.243230 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.246556 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:01.246557 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:01.246557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:19:01.246566 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:01.247695 (+ 1129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:01.248048 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:01.248049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:01.248049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:01.248101 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:01.248271 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:01.248272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:01.248272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:01.248369 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:01.248423 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:01.248638 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 15:19:01.248706 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:19:01.248915 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.238663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43239 (local address 127.31.12.67:43331)
0924 15:19:01.238809 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:01.238846 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:01.239290 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:01.239547 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:01.239550 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:01.239832 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:01.240102 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.240109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.241301 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.241303 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.242661 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.242663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.243317 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.243319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.243480 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.244547 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:01.244559 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:01.246466 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:19:01.247761 (+ 1295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:01.247767 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:01.247769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:01.247979 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:01.248145 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:01.248145 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:01.248146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:01.248226 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:01.248421 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:19:01.248426 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:19:01.248540 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 15:19:01.248619 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 15:19:01.248736 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
I20260924 15:19:01.249364 7453 heartbeater.cc:347] Connected to a master server at 127.31.12.126:43239
I20260924 15:19:01.249478 7453 heartbeater.cc:464] Registering TS with master...
I20260924 15:19:01.249657 7453 heartbeater.cc:511] Master 127.31.12.126:43239 requested a full tablet report, sending...
I20260924 15:19:01.250021 6966 ts_manager.cc:194] Registered new tserver with Master: c97581888e264d1fa8711d8c8b66736a (127.31.12.67:37441)
I20260924 15:19:01.250440 6966 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:43331
I20260924 15:19:01.253432 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 7320
I20260924 15:19:01.253607 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:19:01.253839 31793 external_mini_cluster.cc:1428] Setting key 50305e7a712f7d737095232efe7eb6d2
I20260924 15:19:01.254710 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:19:01.256259 7459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.250383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43239 (local address 127.31.12.67:42393)
0924 15:19:01.250519 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:01.250531 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:01.250588 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:01.250727 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:01.250728 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:01.250903 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:01.250956 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.250957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.251794 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.251795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.252303 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.252304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.252691 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.252691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.252753 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.253211 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:01.253216 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:01.253630 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:19:01.255389 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:01.255390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:01.255390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:01.255552 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:01.255742 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:01.255742 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:01.255743 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:01.255791 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:01.256044 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:19:01.256045 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:19:01.256077 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 15:19:01.256123 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 15:19:01.256193 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260924 15:19:01.256354 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.250457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:42393 (local address 127.31.12.126:43239)
0924 15:19:01.250502 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:01.250504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:01.250578 (+ 74us) server_negotiation.cc:407] Connection header received
0924 15:19:01.250660 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:01.250661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:01.250671 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:01.250723 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:01.251025 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.251716 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.252371 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.252624 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.252690 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.253732 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:01.253733 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:01.253734 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:19:01.253741 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:01.255305 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:01.255618 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:01.255620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:01.255620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:01.255677 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:01.255848 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:01.255849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:01.255850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:01.255981 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:01.256043 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:01.256149 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 15:19:01.256224 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 15:19:01 dist-test-slave-zwbb krb5kdc[6921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 15:19:01 dist-test-slave-zwbb krb5kdc[6921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:01 dist-test-slave-zwbb krb5kdc[6921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 15:19:01.279599 7017 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:01.272525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35010 (local address 127.31.12.126:43239)
0924 15:19:01.272585 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:01.272587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:01.272621 (+ 34us) server_negotiation.cc:407] Connection header received
0924 15:19:01.272707 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:01.272708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:01.272718 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:01.272765 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:01.273588 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.274255 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.275196 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.275518 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.275586 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.279501 (+ 3915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35010: BlockingRecv error: recv got EOF from 127.0.0.1:35010 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 15:19:01 dist-test-slave-zwbb krb5kdc[6921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263141, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM
I20260924 15:19:01.292080 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.285487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35018 (local address 127.31.12.126:43239)
0924 15:19:01.285533 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:01.285535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:01.285615 (+ 80us) server_negotiation.cc:407] Connection header received
0924 15:19:01.285655 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:01.285656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:01.285664 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:01.285734 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:01.286198 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.286666 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.287322 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.287514 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.287594 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.290108 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:01.290109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:01.290110 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:19:01.290116 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:01.291240 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:01.291531 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:01.291531 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:01.291532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:01.291579 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:01.291730 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:01.291730 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:01.291731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:01.291822 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:01.291893 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:01.291968 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 15:19:01.292021 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 15:19:01.293571 6966 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:19:01.295156 6966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:19:01.301918 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.297451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:34463 (local address 127.0.0.1:46380)
0924 15:19:01.297748 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:01.297790 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:01.297844 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:01.298109 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:01.298112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:01.298114 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:19:01.298177 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.298179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.299607 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.299608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.300679 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.300681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.301617 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.301618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.301705 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.301721 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 15:19:01.301790 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 15:19:01.301843 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":23,"thread_start_us":151,"threads_started":1}
I20260924 15:19:01.302060 7480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.297517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:35055 (local address 127.0.0.1:33884)
0924 15:19:01.297854 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:01.297863 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:01.297907 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:01.298611 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:01.298611 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:01.298613 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:19:01.298651 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.298652 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.299557 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.299558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.300836 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.300839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.301800 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.301801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.301862 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.301865 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:19:01.301931 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 15:19:01.301974 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":49,"thread_start_us":58,"threads_started":1}
I20260924 15:19:01.302152 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.297736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:37441 (local address 127.0.0.1:53908)
0924 15:19:01.298210 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:01.298220 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:01.298277 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:01.298791 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:01.298792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:01.298794 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:19:01.298822 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.298823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.299982 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.299983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.301036 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.301037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.301879 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.301879 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.301936 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.301940 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:19:01.302004 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 15:19:01.302049 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"mutex_wait_us":1,"thread_start_us":84,"threads_started":1}
I20260924 15:19:01.302242 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.297856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.31.12.67:37441)
0924 15:19:01.298108 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:01.298113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:01.298268 (+ 155us) server_negotiation.cc:407] Connection header received
0924 15:19:01.298693 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:01.298702 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:01.298717 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:01.298784 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:19:01.298874 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.299903 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.301115 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.301705 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.301774 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.301846 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:01.302101 (+ 255us) server_negotiation.cc:299] Negotiation successful
0924 15:19:01.302141 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":94,"threads_started":1}
I20260924 15:19:01.302400 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.297571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33884 (local address 127.31.12.66:35055)
0924 15:19:01.298461 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:01.298466 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:01.298476 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:19:01.298511 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:01.298513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:01.298528 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:01.298607 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:19:01.298730 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.299466 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.300925 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.301632 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.301694 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.301757 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:01.302248 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 15:19:01.302304 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":80,"threads_started":1}
I20260924 15:19:01.302670 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.297584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46380 (local address 127.31.12.65:34463)
0924 15:19:01.297929 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:01.297935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:01.297947 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:19:01.297989 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:01.297991 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:01.298008 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:01.298091 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:19:01.298370 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.299501 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.300768 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.301534 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.301614 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.301694 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:01.301804 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 15:19:01.301841 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":70,"threads_started":1}
I20260924 15:19:01.302873 7386 tablet_service.cc:1467] Processing CreateTablet for tablet 2274a42ed34241f5ad64d7832678af2a (DEFAULT_TABLE table=test-table [id=84e876fe6d3343c6a81b37af6e4b3ad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:19:01.303010 7245 tablet_service.cc:1467] Processing CreateTablet for tablet 2274a42ed34241f5ad64d7832678af2a (DEFAULT_TABLE table=test-table [id=84e876fe6d3343c6a81b37af6e4b3ad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:19:01.303224 7386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2274a42ed34241f5ad64d7832678af2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:01.303370 7245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2274a42ed34241f5ad64d7832678af2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:01.305893 7104 tablet_service.cc:1467] Processing CreateTablet for tablet 2274a42ed34241f5ad64d7832678af2a (DEFAULT_TABLE table=test-table [id=84e876fe6d3343c6a81b37af6e4b3ad7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:19:01.306231 7104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2274a42ed34241f5ad64d7832678af2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:01.306331 7482 tablet_bootstrap.cc:492] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a: Bootstrap starting.
I20260924 15:19:01.306916 7482 tablet_bootstrap.cc:654] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:01.307200 7482 log.cc:824] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a: Log is configured to *not* fsync() on all Append() calls
I20260924 15:19:01.307547 7483 tablet_bootstrap.cc:492] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4: Bootstrap starting.
I20260924 15:19:01.307904 7482 tablet_bootstrap.cc:492] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a: No bootstrap required, opened a new log
I20260924 15:19:01.307955 7482 ts_tablet_manager.cc:1397] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:19:01.309420 7485 tablet_bootstrap.cc:492] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0: Bootstrap starting.
I20260924 15:19:01.309700 7482 raft_consensus.cc:348] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.309875 7482 raft_consensus.cc:374] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:01.309909 7482 raft_consensus.cc:729] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97581888e264d1fa8711d8c8b66736a, State: Initialized, Role: FOLLOWER
I20260924 15:19:01.310075 7485 tablet_bootstrap.cc:654] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:01.310036 7482 consensus_queue.cc:260] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.310331 7482 ts_tablet_manager.cc:1428] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 15:19:01.310354 7453 heartbeater.cc:503] Master 127.31.12.126:43239 was elected leader, sending a full tablet report...
I20260924 15:19:01.310400 7485 log.cc:824] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0: Log is configured to *not* fsync() on all Append() calls
I20260924 15:19:01.310976 7483 tablet_bootstrap.cc:654] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:01.311257 7485 tablet_bootstrap.cc:492] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0: No bootstrap required, opened a new log
I20260924 15:19:01.311291 7483 log.cc:824] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4: Log is configured to *not* fsync() on all Append() calls
I20260924 15:19:01.311327 7485 ts_tablet_manager.cc:1397] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 15:19:01.312036 7483 tablet_bootstrap.cc:492] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4: No bootstrap required, opened a new log
I20260924 15:19:01.312104 7483 ts_tablet_manager.cc:1397] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 15:19:01.313339 7485 raft_consensus.cc:348] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.313529 7485 raft_consensus.cc:374] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:01.313566 7485 raft_consensus.cc:729] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0510948451454cf994e41c6f5390beb0, State: Initialized, Role: FOLLOWER
I20260924 15:19:01.313678 7485 consensus_queue.cc:260] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.313937 7485 ts_tablet_manager.cc:1428] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 15:19:01.313987 7171 heartbeater.cc:503] Master 127.31.12.126:43239 was elected leader, sending a full tablet report...
I20260924 15:19:01.314106 7483 raft_consensus.cc:348] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.314252 7483 raft_consensus.cc:374] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:01.314292 7483 raft_consensus.cc:729] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82056c4aa10a4f63b50c264c3e62a0a4, State: Initialized, Role: FOLLOWER
I20260924 15:19:01.314402 7483 consensus_queue.cc:260] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.314687 7311 heartbeater.cc:503] Master 127.31.12.126:43239 was elected leader, sending a full tablet report...
I20260924 15:19:01.314685 7483 ts_tablet_manager.cc:1428] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260924 15:19:01.344703 7312 tablet.cc:2369] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:19:01.428407 7489 raft_consensus.cc:482] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:19:01.428537 7489 raft_consensus.cc:504] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.428973 7489 leader_election.cc:288] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82056c4aa10a4f63b50c264c3e62a0a4 (127.31.12.66:35055), c97581888e264d1fa8711d8c8b66736a (127.31.12.67:37441)
I20260924 15:19:01.429860 7490 raft_consensus.cc:482] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:19:01.429975 7490 raft_consensus.cc:504] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.430305 7490 leader_election.cc:288] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0510948451454cf994e41c6f5390beb0 (127.31.12.65:34463), c97581888e264d1fa8711d8c8b66736a (127.31.12.67:37441)
I20260924 15:19:01.432816 7481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.429412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:48001 (local address 127.31.12.67:37441)
0924 15:19:01.429478 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:01.429480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:01.429574 (+ 94us) server_negotiation.cc:407] Connection header received
0924 15:19:01.429623 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:01.429623 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:01.429634 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:01.429690 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:19:01.429864 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.430534 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.431895 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.432435 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.432481 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.432490 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:01.432714 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 15:19:01.432751 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:19:01.432838 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.429204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:35055 (local address 127.31.12.65:53139)
0924 15:19:01.429272 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:01.429288 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:01.429369 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:01.429596 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:01.429598 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:01.429600 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:19:01.429683 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.429685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.430550 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.430550 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.431781 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.431784 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.432577 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.432578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.432663 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.432666 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:19:01.432709 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 15:19:01.432749 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 15:19:01.433046 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.429319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:37441 (local address 127.31.12.65:48001)
0924 15:19:01.429497 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:01.429507 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:01.429565 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:01.429703 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:01.429704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:01.429705 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:19:01.429799 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.429801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.430624 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.430625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.431805 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.431807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.432559 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.432560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.432641 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.432644 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:19:01.432921 (+ 277us) client_negotiation.cc:240] Negotiation successful
0924 15:19:01.432958 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":88,"threads_started":1}
I20260924 15:19:01.433103 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.429299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:53139 (local address 127.31.12.66:35055)
0924 15:19:01.429347 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:01.429349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:01.429360 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:19:01.429496 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:01.429497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:01.429509 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:01.429575 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:19:01.429764 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.430455 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.431884 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.432470 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.432538 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.432545 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:01.433008 (+ 463us) server_negotiation.cc:299] Negotiation successful
0924 15:19:01.433042 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":27}
I20260924 15:19:01.433548 7265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2274a42ed34241f5ad64d7832678af2a" candidate_uuid: "0510948451454cf994e41c6f5390beb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" is_pre_election: true
I20260924 15:19:01.433655 7265 raft_consensus.cc:2454] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0510948451454cf994e41c6f5390beb0 in term 0.
I20260924 15:19:01.433939 7057 leader_election.cc:302] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0510948451454cf994e41c6f5390beb0, 82056c4aa10a4f63b50c264c3e62a0a4; no voters:
I20260924 15:19:01.433930 7406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2274a42ed34241f5ad64d7832678af2a" candidate_uuid: "0510948451454cf994e41c6f5390beb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97581888e264d1fa8711d8c8b66736a" is_pre_election: true
I20260924 15:19:01.434058 7406 raft_consensus.cc:2454] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0510948451454cf994e41c6f5390beb0 in term 0.
I20260924 15:19:01.434072 7489 raft_consensus.cc:2781] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:19:01.434118 7489 raft_consensus.cc:482] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:19:01.434140 7489 raft_consensus.cc:3037] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:01.434168 7492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.430474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:37441 (local address 127.31.12.66:38549)
0924 15:19:01.430720 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:01.430730 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:01.430774 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:01.431077 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:01.431078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:01.431079 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:19:01.431144 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.431145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.431984 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.431985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.433131 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.433133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.433900 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.433901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.433985 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.433989 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:19:01.434037 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 15:19:01.434076 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":83,"threads_started":1}
I20260924 15:19:01.434468 7493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.430685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:38549 (local address 127.31.12.67:37441)
0924 15:19:01.430934 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:01.430936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:01.430942 (+ 6us) server_negotiation.cc:407] Connection header received
0924 15:19:01.430960 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:01.430961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:01.430969 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:01.431200 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:19:01.431228 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.431902 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.433227 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.433804 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.434326 (+ 522us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.434335 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:01.434354 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 15:19:01.434388 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":89,"threads_started":1}
I20260924 15:19:01.434523 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.430426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:34463 (local address 127.31.12.66:54401)
0924 15:19:01.430474 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:01.430484 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:01.430516 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:01.431062 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:01.431063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:01.431065 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:19:01.431114 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.431115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.431884 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.431885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.433086 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.433088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.434260 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.434261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.434358 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.434362 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:19:01.434410 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 15:19:01.434449 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":86}
I20260924 15:19:01.434777 7406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2274a42ed34241f5ad64d7832678af2a" candidate_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97581888e264d1fa8711d8c8b66736a" is_pre_election: true
I20260924 15:19:01.434880 7406 raft_consensus.cc:2454] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82056c4aa10a4f63b50c264c3e62a0a4 in term 0.
I20260924 15:19:01.435071 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.430858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:54401 (local address 127.31.12.65:34463)
0924 15:19:01.430931 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:01.430933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:01.430942 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:19:01.430961 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:01.430962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:01.430971 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:01.431023 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:19:01.431176 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.431808 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.433627 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.434155 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.434218 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.434226 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:01.434977 (+ 751us) server_negotiation.cc:299] Negotiation successful
0924 15:19:01.435018 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 15:19:01.435067 7198 leader_election.cc:302] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82056c4aa10a4f63b50c264c3e62a0a4, c97581888e264d1fa8711d8c8b66736a; no voters:
I20260924 15:19:01.435191 7490 raft_consensus.cc:2781] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:19:01.435246 7490 raft_consensus.cc:482] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:19:01.435282 7490 raft_consensus.cc:3037] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:01.435364 7124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2274a42ed34241f5ad64d7832678af2a" candidate_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0510948451454cf994e41c6f5390beb0" is_pre_election: true
I20260924 15:19:01.435366 7489 raft_consensus.cc:504] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.435551 7124 raft_consensus.cc:2379] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 82056c4aa10a4f63b50c264c3e62a0a4 in current term 1: Already voted for candidate 0510948451454cf994e41c6f5390beb0 in this term.
I20260924 15:19:01.435561 7489 leader_election.cc:288] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [CANDIDATE]: Term 1 election: Requested vote from peers 82056c4aa10a4f63b50c264c3e62a0a4 (127.31.12.66:35055), c97581888e264d1fa8711d8c8b66736a (127.31.12.67:37441)
I20260924 15:19:01.435865 7265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2274a42ed34241f5ad64d7832678af2a" candidate_uuid: "0510948451454cf994e41c6f5390beb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82056c4aa10a4f63b50c264c3e62a0a4"
I20260924 15:19:01.435865 7406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2274a42ed34241f5ad64d7832678af2a" candidate_uuid: "0510948451454cf994e41c6f5390beb0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97581888e264d1fa8711d8c8b66736a"
I20260924 15:19:01.435945 7406 raft_consensus.cc:3037] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:01.436100 7490 raft_consensus.cc:504] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.436206 7265 raft_consensus.cc:2379] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0510948451454cf994e41c6f5390beb0 in current term 1: Already voted for candidate 82056c4aa10a4f63b50c264c3e62a0a4 in this term.
I20260924 15:19:01.436235 7490 leader_election.cc:288] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4 [CANDIDATE]: Term 1 election: Requested vote from peers 0510948451454cf994e41c6f5390beb0 (127.31.12.65:34463), c97581888e264d1fa8711d8c8b66736a (127.31.12.67:37441)
I20260924 15:19:01.436439 7124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2274a42ed34241f5ad64d7832678af2a" candidate_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0510948451454cf994e41c6f5390beb0"
I20260924 15:19:01.436430 7405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2274a42ed34241f5ad64d7832678af2a" candidate_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97581888e264d1fa8711d8c8b66736a"
I20260924 15:19:01.436527 7124 raft_consensus.cc:2379] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 82056c4aa10a4f63b50c264c3e62a0a4 in current term 1: Already voted for candidate 0510948451454cf994e41c6f5390beb0 in this term.
I20260924 15:19:01.436718 7199 leader_election.cc:302] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 82056c4aa10a4f63b50c264c3e62a0a4; no voters: 0510948451454cf994e41c6f5390beb0, c97581888e264d1fa8711d8c8b66736a
I20260924 15:19:01.436825 7490 raft_consensus.cc:2726] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260924 15:19:01.437124 7406 raft_consensus.cc:2454] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0510948451454cf994e41c6f5390beb0 in term 1.
I20260924 15:19:01.437314 7057 leader_election.cc:302] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0510948451454cf994e41c6f5390beb0, c97581888e264d1fa8711d8c8b66736a; no voters: 82056c4aa10a4f63b50c264c3e62a0a4
I20260924 15:19:01.437430 7489 raft_consensus.cc:2781] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:19:01.437609 7489 raft_consensus.cc:686] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [term 1 LEADER]: Becoming Leader. State: Replica: 0510948451454cf994e41c6f5390beb0, State: Running, Role: LEADER
I20260924 15:19:01.437721 7489 consensus_queue.cc:237] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.438536 6966 catalog_manager.cc:5495] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0510948451454cf994e41c6f5390beb0 (127.31.12.65). New cstate: current_term: 1 leader_uuid: "0510948451454cf994e41c6f5390beb0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } health_report { overall_health: UNKNOWN } } }
W20260924 15:19:01.439826 7172 tablet.cc:2369] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:19:01.444957 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.443000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46386 (local address 127.31.12.65:34463)
0924 15:19:01.443035 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:01.443037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:01.443068 (+ 31us) server_negotiation.cc:407] Connection header received
0924 15:19:01.443095 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:01.443095 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:01.443109 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:01.443158 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:19:01.443245 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.443678 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.444352 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.444525 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.444574 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.444693 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:19:01.444781 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:19:01.444836 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:01.444885 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 15:19:01.444900 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:19:01.446540 7406 raft_consensus.cc:1261] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a [term 1 FOLLOWER]: Refusing update from remote peer 0510948451454cf994e41c6f5390beb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:19:01.446578 7265 raft_consensus.cc:1261] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 FOLLOWER]: Refusing update from remote peer 0510948451454cf994e41c6f5390beb0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:19:01.446797 7494 consensus_queue.cc:1035] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:19:01.446933 7489 consensus_queue.cc:1035] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:19:01.449617 7498 mvcc.cc:205] Tried to move back new op lower bound from 7332917827362816000 to 7332917827329613824. Current Snapshot: MvccSnapshot[applied={T|T < 7332917827362816000}]
I20260924 15:19:01.449801 7497 mvcc.cc:205] Tried to move back new op lower bound from 7332917827362816000 to 7332917827329613824. Current Snapshot: MvccSnapshot[applied={T|T < 7332917827362816000 or (T in {7332917827362816000})}]
I20260924 15:19:01.450029 7499 mvcc.cc:205] Tried to move back new op lower bound from 7332917827362816000 to 7332917827329613824. Current Snapshot: MvccSnapshot[applied={T|T < 7332917827362816000 or (T in {7332917827362816000})}]
I20260924 15:19:01.451087 6965 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:19:01.451177 6965 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:19:01.451939 6965 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974: Sending DeleteTablet for 3 replicas of tablet 2274a42ed34241f5ad64d7832678af2a
I20260924 15:19:01.452301 7104 tablet_service.cc:1514] Processing DeleteTablet for tablet 2274a42ed34241f5ad64d7832678af2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:19:01 UTC) from {username='oryx', principal='oryx/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:46380
I20260924 15:19:01.452391 7386 tablet_service.cc:1514] Processing DeleteTablet for tablet 2274a42ed34241f5ad64d7832678af2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:19:01 UTC) from {username='oryx', principal='oryx/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:53908
I20260924 15:19:01.452504 7507 tablet_replica.cc:330] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0: stopping tablet replica
I20260924 15:19:01.452606 7507 raft_consensus.cc:2229] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:19:01.452643 7245 tablet_service.cc:1514] Processing DeleteTablet for tablet 2274a42ed34241f5ad64d7832678af2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:19:01 UTC) from {username='oryx', principal='oryx/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:33884
I20260924 15:19:01.452629 7508 tablet_replica.cc:330] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a: stopping tablet replica
I20260924 15:19:01.452701 7507 raft_consensus.cc:2258] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:19:01.452831 6965 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:19:01.452987 6965 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:19:01.452896 7508 raft_consensus.cc:2229] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:19:01.453197 7509 tablet_replica.cc:330] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4: stopping tablet replica
I20260924 15:19:01.453292 7509 raft_consensus.cc:2229] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:19:01.453402 7509 raft_consensus.cc:2258] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:19:01.453408 7508 raft_consensus.cc:2258] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:19:01.453989 7508 ts_tablet_manager.cc:1905] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:19:01.453995 7509 ts_tablet_manager.cc:1905] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:19:01.454588 7245 tablet_service.cc:1467] Processing CreateTablet for tablet 77a2fc42fe194514a812efc6770a20b0 (DEFAULT_TABLE table=test-table [id=ff177e56e32d45ddb46f670e68e1142f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:19:01.454638 7104 tablet_service.cc:1467] Processing CreateTablet for tablet 77a2fc42fe194514a812efc6770a20b0 (DEFAULT_TABLE table=test-table [id=ff177e56e32d45ddb46f670e68e1142f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:19:01.454702 7245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a2fc42fe194514a812efc6770a20b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:01.454753 7104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a2fc42fe194514a812efc6770a20b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:01.455932 7508 ts_tablet_manager.cc:1918] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:19:01.456001 7386 tablet_service.cc:1467] Processing CreateTablet for tablet 77a2fc42fe194514a812efc6770a20b0 (DEFAULT_TABLE table=test-table [id=ff177e56e32d45ddb46f670e68e1142f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:19:01.456117 7386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a2fc42fe194514a812efc6770a20b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:01.456246 7483 tablet_bootstrap.cc:492] T 77a2fc42fe194514a812efc6770a20b0 P 82056c4aa10a4f63b50c264c3e62a0a4: Bootstrap starting.
I20260924 15:19:01.456632 7483 tablet_bootstrap.cc:654] T 77a2fc42fe194514a812efc6770a20b0 P 82056c4aa10a4f63b50c264c3e62a0a4: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:01.456622 7507 ts_tablet_manager.cc:1905] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:19:01.456920 7485 tablet_bootstrap.cc:492] T 77a2fc42fe194514a812efc6770a20b0 P 0510948451454cf994e41c6f5390beb0: Bootstrap starting.
I20260924 15:19:01.457309 7485 tablet_bootstrap.cc:654] T 77a2fc42fe194514a812efc6770a20b0 P 0510948451454cf994e41c6f5390beb0: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:01.457333 7482 tablet_bootstrap.cc:492] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a: Bootstrap starting.
I20260924 15:19:01.456103 7508 log.cc:1196] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/2274a42ed34241f5ad64d7832678af2a
I20260924 15:19:01.457763 7482 tablet_bootstrap.cc:654] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:01.457787 7508 ts_tablet_manager.cc:1939] T 2274a42ed34241f5ad64d7832678af2a P c97581888e264d1fa8711d8c8b66736a: Deleting consensus metadata
I20260924 15:19:01.458385 6952 catalog_manager.cc:4841] TS c97581888e264d1fa8711d8c8b66736a (127.31.12.67:37441): tablet 2274a42ed34241f5ad64d7832678af2a (table test-table [id=84e876fe6d3343c6a81b37af6e4b3ad7]) successfully deleted
I20260924 15:19:01.458704 7507 ts_tablet_manager.cc:1918] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:19:01.458791 7507 log.cc:1196] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/wal/wals/2274a42ed34241f5ad64d7832678af2a
I20260924 15:19:01.458791 7509 ts_tablet_manager.cc:1918] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:19:01.458853 7509 log.cc:1196] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/2274a42ed34241f5ad64d7832678af2a
I20260924 15:19:01.459040 7507 ts_tablet_manager.cc:1939] T 2274a42ed34241f5ad64d7832678af2a P 0510948451454cf994e41c6f5390beb0: Deleting consensus metadata
I20260924 15:19:01.459120 7509 ts_tablet_manager.cc:1939] T 2274a42ed34241f5ad64d7832678af2a P 82056c4aa10a4f63b50c264c3e62a0a4: Deleting consensus metadata
I20260924 15:19:01.459457 6953 catalog_manager.cc:4841] TS 0510948451454cf994e41c6f5390beb0 (127.31.12.65:34463): tablet 2274a42ed34241f5ad64d7832678af2a (table test-table [id=84e876fe6d3343c6a81b37af6e4b3ad7]) successfully deleted
I20260924 15:19:01.459717 7483 tablet_bootstrap.cc:492] T 77a2fc42fe194514a812efc6770a20b0 P 82056c4aa10a4f63b50c264c3e62a0a4: No bootstrap required, opened a new log
I20260924 15:19:01.459771 7483 ts_tablet_manager.cc:1397] T 77a2fc42fe194514a812efc6770a20b0 P 82056c4aa10a4f63b50c264c3e62a0a4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 15:19:01.459899 7483 raft_consensus.cc:348] T 77a2fc42fe194514a812efc6770a20b0 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.459968 7483 raft_consensus.cc:374] T 77a2fc42fe194514a812efc6770a20b0 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:01.460001 7483 raft_consensus.cc:729] T 77a2fc42fe194514a812efc6770a20b0 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82056c4aa10a4f63b50c264c3e62a0a4, State: Initialized, Role: FOLLOWER
I20260924 15:19:01.460039 7483 consensus_queue.cc:260] T 77a2fc42fe194514a812efc6770a20b0 P 82056c4aa10a4f63b50c264c3e62a0a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.460129 7483 ts_tablet_manager.cc:1428] T 77a2fc42fe194514a812efc6770a20b0 P 82056c4aa10a4f63b50c264c3e62a0a4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:01.460248 6952 catalog_manager.cc:4841] TS 82056c4aa10a4f63b50c264c3e62a0a4 (127.31.12.66:35055): tablet 2274a42ed34241f5ad64d7832678af2a (table test-table [id=84e876fe6d3343c6a81b37af6e4b3ad7]) successfully deleted
I20260924 15:19:01.460484 7485 tablet_bootstrap.cc:492] T 77a2fc42fe194514a812efc6770a20b0 P 0510948451454cf994e41c6f5390beb0: No bootstrap required, opened a new log
I20260924 15:19:01.460543 7485 ts_tablet_manager.cc:1397] T 77a2fc42fe194514a812efc6770a20b0 P 0510948451454cf994e41c6f5390beb0: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260924 15:19:01.460675 7485 raft_consensus.cc:348] T 77a2fc42fe194514a812efc6770a20b0 P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.460748 7485 raft_consensus.cc:374] T 77a2fc42fe194514a812efc6770a20b0 P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:01.460769 7485 raft_consensus.cc:729] T 77a2fc42fe194514a812efc6770a20b0 P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0510948451454cf994e41c6f5390beb0, State: Initialized, Role: FOLLOWER
I20260924 15:19:01.460814 7485 consensus_queue.cc:260] T 77a2fc42fe194514a812efc6770a20b0 P 0510948451454cf994e41c6f5390beb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.460896 7485 ts_tablet_manager.cc:1428] T 77a2fc42fe194514a812efc6770a20b0 P 0510948451454cf994e41c6f5390beb0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:01.461796 7482 tablet_bootstrap.cc:492] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a: No bootstrap required, opened a new log
I20260924 15:19:01.461860 7482 ts_tablet_manager.cc:1397] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260924 15:19:01.462013 7482 raft_consensus.cc:348] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.462085 7482 raft_consensus.cc:374] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:01.462116 7482 raft_consensus.cc:729] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97581888e264d1fa8711d8c8b66736a, State: Initialized, Role: FOLLOWER
I20260924 15:19:01.462164 7482 consensus_queue.cc:260] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.462247 7482 ts_tablet_manager.cc:1428] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
W20260924 15:19:01.496641 7454 tablet.cc:2369] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:19:01.744261 7486 raft_consensus.cc:482] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:19:01.744354 7486 raft_consensus.cc:504] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.744699 7486 leader_election.cc:288] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0510948451454cf994e41c6f5390beb0 (127.31.12.65:34463), 82056c4aa10a4f63b50c264c3e62a0a4 (127.31.12.66:35055)
I20260924 15:19:01.747735 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.744876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:35055 (local address 127.31.12.67:41669)
0924 15:19:01.744936 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:01.744950 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:01.745014 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:01.745195 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:01.745196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:01.745198 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:19:01.745276 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.745278 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.746053 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.746054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.746877 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.746880 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.747436 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.747437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.747516 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.747521 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:19:01.747599 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 15:19:01.747660 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 15:19:01.747738 7478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.745021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:41669 (local address 127.31.12.66:35055)
0924 15:19:01.745078 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:01.745080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:01.745088 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:19:01.745114 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:01.745115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:01.745129 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:01.745191 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:19:01.745364 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.745984 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.746965 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.747365 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.747420 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.747427 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:01.747607 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 15:19:01.747650 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 15:19:01.747857 7513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.744876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:34463 (local address 127.31.12.67:36257)
0924 15:19:01.745186 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:01.745195 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:01.745243 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:01.745363 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:01.745363 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:01.745365 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:19:01.745431 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.745432 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.746219 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.746220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.747008 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.747010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.747575 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.747576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.747660 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.747664 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:19:01.747722 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 15:19:01.747760 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":175,"thread_start_us":92,"threads_started":1}
I20260924 15:19:01.747871 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.744981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:36257 (local address 127.31.12.65:34463)
0924 15:19:01.745049 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:01.745052 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:01.745244 (+ 192us) server_negotiation.cc:407] Connection header received
0924 15:19:01.745283 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:01.745285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:01.745294 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:01.745351 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:19:01.745535 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.746140 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.747089 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.747479 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.747553 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.747562 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:01.747737 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 15:19:01.747793 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:19:01.748003 7265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a2fc42fe194514a812efc6770a20b0" candidate_uuid: "c97581888e264d1fa8711d8c8b66736a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" is_pre_election: true
I20260924 15:19:01.748035 7124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a2fc42fe194514a812efc6770a20b0" candidate_uuid: "c97581888e264d1fa8711d8c8b66736a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0510948451454cf994e41c6f5390beb0" is_pre_election: true
I20260924 15:19:01.748085 7265 raft_consensus.cc:2454] T 77a2fc42fe194514a812efc6770a20b0 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c97581888e264d1fa8711d8c8b66736a in term 0.
I20260924 15:19:01.748109 7124 raft_consensus.cc:2454] T 77a2fc42fe194514a812efc6770a20b0 P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c97581888e264d1fa8711d8c8b66736a in term 0.
I20260924 15:19:01.748283 7339 leader_election.cc:302] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0510948451454cf994e41c6f5390beb0, c97581888e264d1fa8711d8c8b66736a; no voters:
I20260924 15:19:01.748420 7486 raft_consensus.cc:2781] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:19:01.748469 7486 raft_consensus.cc:482] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:19:01.748485 7486 raft_consensus.cc:3037] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:01.749178 7486 raft_consensus.cc:504] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.749311 7486 leader_election.cc:288] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [CANDIDATE]: Term 1 election: Requested vote from peers 0510948451454cf994e41c6f5390beb0 (127.31.12.65:34463), 82056c4aa10a4f63b50c264c3e62a0a4 (127.31.12.66:35055)
I20260924 15:19:01.749516 7265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a2fc42fe194514a812efc6770a20b0" candidate_uuid: "c97581888e264d1fa8711d8c8b66736a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82056c4aa10a4f63b50c264c3e62a0a4"
I20260924 15:19:01.749600 7124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a2fc42fe194514a812efc6770a20b0" candidate_uuid: "c97581888e264d1fa8711d8c8b66736a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0510948451454cf994e41c6f5390beb0"
I20260924 15:19:01.749645 7265 raft_consensus.cc:3037] T 77a2fc42fe194514a812efc6770a20b0 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:01.749665 7124 raft_consensus.cc:3037] T 77a2fc42fe194514a812efc6770a20b0 P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:01.750466 7265 raft_consensus.cc:2454] T 77a2fc42fe194514a812efc6770a20b0 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c97581888e264d1fa8711d8c8b66736a in term 1.
I20260924 15:19:01.750475 7124 raft_consensus.cc:2454] T 77a2fc42fe194514a812efc6770a20b0 P 0510948451454cf994e41c6f5390beb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c97581888e264d1fa8711d8c8b66736a in term 1.
I20260924 15:19:01.750679 7338 leader_election.cc:302] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82056c4aa10a4f63b50c264c3e62a0a4, c97581888e264d1fa8711d8c8b66736a; no voters:
I20260924 15:19:01.750793 7486 raft_consensus.cc:2781] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:19:01.750916 7486 raft_consensus.cc:686] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [term 1 LEADER]: Becoming Leader. State: Replica: c97581888e264d1fa8711d8c8b66736a, State: Running, Role: LEADER
I20260924 15:19:01.750998 7486 consensus_queue.cc:237] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.751754 6965 catalog_manager.cc:5495] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a reported cstate change: term changed from 0 to 1, leader changed from <none> to c97581888e264d1fa8711d8c8b66736a (127.31.12.67). New cstate: current_term: 1 leader_uuid: "c97581888e264d1fa8711d8c8b66736a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } health_report { overall_health: HEALTHY } } }
Sep 24 15:19:01 dist-test-slave-zwbb krb5kdc[6921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263140, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.126@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM
I20260924 15:19:01.829100 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.822225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.31.12.126:43239)
0924 15:19:01.822395 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:01.822397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:01.822428 (+ 31us) server_negotiation.cc:407] Connection header received
0924 15:19:01.822483 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:01.822483 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:01.822496 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:01.822550 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:01.823032 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.823581 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.824265 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.824452 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.824519 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.826596 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:01.826598 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:01.826598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:19:01.826606 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:01.827889 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:01.828264 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:01.828265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:01.828266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:01.828325 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:01.828523 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:01.828524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:01.828525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:01.828632 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:01.828718 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:01.828872 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 15:19:01.828979 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":100,"threads_started":1}
I20260924 15:19:01.829157 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.822186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43239 (local address 127.0.0.1:35022)
0924 15:19:01.822367 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:01.822378 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:01.822426 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:01.822566 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:01.822566 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:01.822886 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:01.822932 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.822934 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.823659 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.823660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.824179 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.824181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.824536 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.824537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.824594 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.825135 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:01.825140 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:01.826511 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:19:01.827982 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:01.827990 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:01.827993 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:01.828192 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:01.828387 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:01.828387 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:01.828388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:01.828446 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:01.828718 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:19:01.828727 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:19:01.828772 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 15:19:01.828848 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 15:19:01.829038 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":88,"thread_start_us":95,"threads_started":1}
I20260924 15:19:01.831252 6965 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:35022:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 15:19:01.831503 6965 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:19:01.833022 7104 tablet_service.cc:1467] Processing CreateTablet for tablet 7b26bcc4685e40068c8b4d34b1cdd8d8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=461513e53d564d7cb4a4cb9552be276f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:19:01.833041 7245 tablet_service.cc:1467] Processing CreateTablet for tablet 7b26bcc4685e40068c8b4d34b1cdd8d8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=461513e53d564d7cb4a4cb9552be276f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:19:01.833047 7386 tablet_service.cc:1467] Processing CreateTablet for tablet 7b26bcc4685e40068c8b4d34b1cdd8d8 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=461513e53d564d7cb4a4cb9552be276f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 15:19:01.833197 7386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b26bcc4685e40068c8b4d34b1cdd8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:01.833199 7245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b26bcc4685e40068c8b4d34b1cdd8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:01.833197 7104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b26bcc4685e40068c8b4d34b1cdd8d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:01.834417 7485 tablet_bootstrap.cc:492] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 0510948451454cf994e41c6f5390beb0: Bootstrap starting.
I20260924 15:19:01.834733 7482 tablet_bootstrap.cc:492] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a: Bootstrap starting.
I20260924 15:19:01.834731 7483 tablet_bootstrap.cc:492] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 82056c4aa10a4f63b50c264c3e62a0a4: Bootstrap starting.
I20260924 15:19:01.834784 7485 tablet_bootstrap.cc:654] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 0510948451454cf994e41c6f5390beb0: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:01.835130 7482 tablet_bootstrap.cc:654] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:01.835143 7483 tablet_bootstrap.cc:654] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 82056c4aa10a4f63b50c264c3e62a0a4: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:01.835438 7485 tablet_bootstrap.cc:492] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 0510948451454cf994e41c6f5390beb0: No bootstrap required, opened a new log
I20260924 15:19:01.835487 7485 ts_tablet_manager.cc:1397] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 0510948451454cf994e41c6f5390beb0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 15:19:01.835611 7485 raft_consensus.cc:348] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.835704 7485 raft_consensus.cc:374] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:01.835738 7485 raft_consensus.cc:729] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0510948451454cf994e41c6f5390beb0, State: Initialized, Role: FOLLOWER
I20260924 15:19:01.835783 7485 consensus_queue.cc:260] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 0510948451454cf994e41c6f5390beb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.835803 7482 tablet_bootstrap.cc:492] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a: No bootstrap required, opened a new log
I20260924 15:19:01.835846 7482 ts_tablet_manager.cc:1397] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 15:19:01.835884 7485 ts_tablet_manager.cc:1428] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 0510948451454cf994e41c6f5390beb0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:01.835930 7502 tablet_replica.cc:439] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 0510948451454cf994e41c6f5390beb0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } } }
I20260924 15:19:01.836058 7502 tablet_replica.cc:442] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 0510948451454cf994e41c6f5390beb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:19:01.836148 7482 raft_consensus.cc:348] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.836228 7482 raft_consensus.cc:374] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:01.836262 7482 raft_consensus.cc:729] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97581888e264d1fa8711d8c8b66736a, State: Initialized, Role: FOLLOWER
I20260924 15:19:01.836302 7482 consensus_queue.cc:260] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.836351 7483 tablet_bootstrap.cc:492] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 82056c4aa10a4f63b50c264c3e62a0a4: No bootstrap required, opened a new log
I20260924 15:19:01.836380 7482 ts_tablet_manager.cc:1428] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 15:19:01.836392 7483 ts_tablet_manager.cc:1397] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 82056c4aa10a4f63b50c264c3e62a0a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:19:01.836506 7486 tablet_replica.cc:439] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } } }
I20260924 15:19:01.836539 7483 raft_consensus.cc:348] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.836589 7486 tablet_replica.cc:442] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:19:01.836611 7483 raft_consensus.cc:374] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:01.836635 7483 raft_consensus.cc:729] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82056c4aa10a4f63b50c264c3e62a0a4, State: Initialized, Role: FOLLOWER
I20260924 15:19:01.836688 7483 consensus_queue.cc:260] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 82056c4aa10a4f63b50c264c3e62a0a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.836787 7483 ts_tablet_manager.cc:1428] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 82056c4aa10a4f63b50c264c3e62a0a4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:01.836854 7490 tablet_replica.cc:439] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 82056c4aa10a4f63b50c264c3e62a0a4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } } }
I20260924 15:19:01.836949 7490 tablet_replica.cc:442] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 82056c4aa10a4f63b50c264c3e62a0a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:19:01.970801 7486 raft_consensus.cc:482] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:19:01.971032 7486 raft_consensus.cc:504] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.971220 7486 leader_election.cc:288] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0510948451454cf994e41c6f5390beb0 (127.31.12.65:34463), 82056c4aa10a4f63b50c264c3e62a0a4 (127.31.12.66:35055)
I20260924 15:19:01.971500 7265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26bcc4685e40068c8b4d34b1cdd8d8" candidate_uuid: "c97581888e264d1fa8711d8c8b66736a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" is_pre_election: true
I20260924 15:19:01.971494 7124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26bcc4685e40068c8b4d34b1cdd8d8" candidate_uuid: "c97581888e264d1fa8711d8c8b66736a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0510948451454cf994e41c6f5390beb0" is_pre_election: true
I20260924 15:19:01.971607 7265 raft_consensus.cc:2454] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c97581888e264d1fa8711d8c8b66736a in term 0.
I20260924 15:19:01.971607 7124 raft_consensus.cc:2454] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c97581888e264d1fa8711d8c8b66736a in term 0.
I20260924 15:19:01.971774 7338 leader_election.cc:302] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82056c4aa10a4f63b50c264c3e62a0a4, c97581888e264d1fa8711d8c8b66736a; no voters:
I20260924 15:19:01.971915 7486 raft_consensus.cc:2781] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:19:01.971956 7486 raft_consensus.cc:482] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:19:01.971987 7486 raft_consensus.cc:3037] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:01.972652 7486 raft_consensus.cc:504] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.972790 7486 leader_election.cc:288] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a [CANDIDATE]: Term 1 election: Requested vote from peers 0510948451454cf994e41c6f5390beb0 (127.31.12.65:34463), 82056c4aa10a4f63b50c264c3e62a0a4 (127.31.12.66:35055)
I20260924 15:19:01.972959 7124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26bcc4685e40068c8b4d34b1cdd8d8" candidate_uuid: "c97581888e264d1fa8711d8c8b66736a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0510948451454cf994e41c6f5390beb0"
I20260924 15:19:01.973079 7124 raft_consensus.cc:3037] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:01.973071 7265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b26bcc4685e40068c8b4d34b1cdd8d8" candidate_uuid: "c97581888e264d1fa8711d8c8b66736a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82056c4aa10a4f63b50c264c3e62a0a4"
I20260924 15:19:01.973150 7265 raft_consensus.cc:3037] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:01.973953 7265 raft_consensus.cc:2454] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c97581888e264d1fa8711d8c8b66736a in term 1.
I20260924 15:19:01.973953 7124 raft_consensus.cc:2454] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 0510948451454cf994e41c6f5390beb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c97581888e264d1fa8711d8c8b66736a in term 1.
I20260924 15:19:01.974153 7339 leader_election.cc:302] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0510948451454cf994e41c6f5390beb0, c97581888e264d1fa8711d8c8b66736a; no voters:
I20260924 15:19:01.974262 7486 raft_consensus.cc:2781] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:19:01.974349 7486 raft_consensus.cc:686] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a [term 1 LEADER]: Becoming Leader. State: Replica: c97581888e264d1fa8711d8c8b66736a, State: Running, Role: LEADER
I20260924 15:19:01.974406 7486 consensus_queue.cc:237] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:01.974692 7514 tablet_replica.cc:439] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a: TxnStatusTablet state changed. Reason: New leader c97581888e264d1fa8711d8c8b66736a. Latest consensus state: current_term: 1 leader_uuid: "c97581888e264d1fa8711d8c8b66736a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } } }
I20260924 15:19:01.974792 7514 tablet_replica.cc:442] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:19:01.974943 7527 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 15:19:01.975100 7527 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 15:19:01.975107 6966 catalog_manager.cc:5495] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a reported cstate change: term changed from 0 to 1, leader changed from <none> to c97581888e264d1fa8711d8c8b66736a (127.31.12.67). New cstate: current_term: 1 leader_uuid: "c97581888e264d1fa8711d8c8b66736a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } health_report { overall_health: HEALTHY } } }
I20260924 15:19:01.984036 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.981130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:37441 (local address 127.0.0.1:53924)
0924 15:19:01.981209 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:01.981224 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:01.981283 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:01.981537 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:01.981538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:01.981540 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:19:01.981603 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.981611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.982355 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.982357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.982998 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.983000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.983409 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.983411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.983479 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.983494 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:19:01.983797 (+ 303us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:19:01.983809 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 15:19:01.983855 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 15:19:01.983922 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 15:19:01.984051 7530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.981204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53924 (local address 127.31.12.67:37441)
0924 15:19:01.981328 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:01.981330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:01.981340 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:19:01.981386 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:01.981387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:01.981404 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:01.981527 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:19:01.981687 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.982266 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.983091 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.983327 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.983409 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.983594 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:19:01.983710 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:19:01.983801 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:01.983897 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 15:19:01.983920 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":63,"threads_started":1}
I20260924 15:19:01.985625 7265 raft_consensus.cc:1261] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 FOLLOWER]: Refusing update from remote peer c97581888e264d1fa8711d8c8b66736a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:19:01.985729 7124 raft_consensus.cc:1261] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 0510948451454cf994e41c6f5390beb0 [term 1 FOLLOWER]: Refusing update from remote peer c97581888e264d1fa8711d8c8b66736a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:19:01.985916 7486 consensus_queue.cc:1035] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a [LEADER]: Connected to new peer: Peer: permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:19:01.986056 7514 consensus_queue.cc:1035] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:19:01.986663 7490 tablet_replica.cc:439] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 82056c4aa10a4f63b50c264c3e62a0a4: TxnStatusTablet state changed. Reason: New leader c97581888e264d1fa8711d8c8b66736a. Latest consensus state: current_term: 1 leader_uuid: "c97581888e264d1fa8711d8c8b66736a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } } }
I20260924 15:19:01.986760 7490 tablet_replica.cc:442] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 82056c4aa10a4f63b50c264c3e62a0a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:19:01.986712 7502 tablet_replica.cc:439] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 0510948451454cf994e41c6f5390beb0: TxnStatusTablet state changed. Reason: New leader c97581888e264d1fa8711d8c8b66736a. Latest consensus state: current_term: 1 leader_uuid: "c97581888e264d1fa8711d8c8b66736a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } } }
I20260924 15:19:01.986814 7502 tablet_replica.cc:442] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 0510948451454cf994e41c6f5390beb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:19:01.987545 7486 tablet_replica.cc:439] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c97581888e264d1fa8711d8c8b66736a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } } }
I20260924 15:19:01.987630 7486 tablet_replica.cc:442] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:19:01.987579 7502 tablet_replica.cc:439] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 0510948451454cf994e41c6f5390beb0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c97581888e264d1fa8711d8c8b66736a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } } }
I20260924 15:19:01.987656 7502 tablet_replica.cc:442] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 0510948451454cf994e41c6f5390beb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:19:01.987843 7486 tablet_replica.cc:439] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c97581888e264d1fa8711d8c8b66736a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } } }
I20260924 15:19:01.987936 7486 tablet_replica.cc:442] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P c97581888e264d1fa8711d8c8b66736a: This TxnStatusTablet replica's current role is: LEADER
I20260924 15:19:01.988185 7490 tablet_replica.cc:439] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 82056c4aa10a4f63b50c264c3e62a0a4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c97581888e264d1fa8711d8c8b66736a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } } }
I20260924 15:19:01.988262 7490 tablet_replica.cc:442] T 7b26bcc4685e40068c8b4d34b1cdd8d8 P 82056c4aa10a4f63b50c264c3e62a0a4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 15:19:01.993520 7530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.990860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53926 (local address 127.31.12.67:37441)
0924 15:19:01.990900 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:01.990902 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:01.990968 (+ 66us) server_negotiation.cc:407] Connection header received
0924 15:19:01.991037 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:01.991038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:01.991047 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:01.991103 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:19:01.991230 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.991722 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.992831 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.993040 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.993100 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.993263 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:19:01.993289 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:19:01.993325 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:01.993386 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 15:19:01.993404 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:19:01.999110 7530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.996079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:59593 (local address 127.31.12.67:37441)
0924 15:19:01.996121 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:01.996124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:01.996247 (+ 123us) server_negotiation.cc:407] Connection header received
0924 15:19:01.996341 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:01.996342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:01.996350 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:01.996392 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:19:01.996543 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.997075 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.998311 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.998508 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.998568 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.998773 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:19:01.998802 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:19:01.998861 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:01.999034 (+ 173us) server_negotiation.cc:299] Negotiation successful
0924 15:19:01.999053 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:19:01.999265 7543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:01.996034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:37441 (local address 127.31.12.67:59593)
0924 15:19:01.996174 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:01.996185 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:01.996281 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:01.996407 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:01.996407 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:01.996409 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:19:01.996494 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.996495 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.997144 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.997145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.998237 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:01.998240 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:01.998580 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:01.998581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:01.998646 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:01.998667 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:19:01.998877 (+ 210us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:19:01.998879 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:19:01.999014 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 15:19:01.999061 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":85,"threads_started":1}
I20260924 15:19:02.001703 7124 raft_consensus.cc:1261] T 77a2fc42fe194514a812efc6770a20b0 P 0510948451454cf994e41c6f5390beb0 [term 1 FOLLOWER]: Refusing update from remote peer c97581888e264d1fa8711d8c8b66736a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:19:02.001718 7265 raft_consensus.cc:1261] T 77a2fc42fe194514a812efc6770a20b0 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 FOLLOWER]: Refusing update from remote peer c97581888e264d1fa8711d8c8b66736a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:19:02.001920 7486 consensus_queue.cc:1035] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:19:02.002013 7514 consensus_queue.cc:1035] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [LEADER]: Connected to new peer: Peer: permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:19:02.013512 6966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:19:02.013622 6966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:19:02.014243 6966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974: Sending DeleteTablet for 3 replicas of tablet 77a2fc42fe194514a812efc6770a20b0
I20260924 15:19:02.014693 7245 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a2fc42fe194514a812efc6770a20b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:19:02 UTC) from {username='oryx', principal='oryx/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:33884
I20260924 15:19:02.014755 7104 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a2fc42fe194514a812efc6770a20b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:19:02 UTC) from {username='oryx', principal='oryx/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:46380
I20260924 15:19:02.014990 7548 tablet_replica.cc:330] T 77a2fc42fe194514a812efc6770a20b0 P 82056c4aa10a4f63b50c264c3e62a0a4: stopping tablet replica
I20260924 15:19:02.015053 7548 raft_consensus.cc:2229] T 77a2fc42fe194514a812efc6770a20b0 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:19:02.015122 7548 raft_consensus.cc:2258] T 77a2fc42fe194514a812efc6770a20b0 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:19:02.015146 7386 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a2fc42fe194514a812efc6770a20b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:19:02 UTC) from {username='oryx', principal='oryx/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:53908
I20260924 15:19:02.015439 7548 ts_tablet_manager.cc:1905] T 77a2fc42fe194514a812efc6770a20b0 P 82056c4aa10a4f63b50c264c3e62a0a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:19:02.015933 7549 tablet_replica.cc:330] T 77a2fc42fe194514a812efc6770a20b0 P 0510948451454cf994e41c6f5390beb0: stopping tablet replica
I20260924 15:19:02.016971 7550 tablet_replica.cc:330] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a: stopping tablet replica
I20260924 15:19:02.017022 7550 raft_consensus.cc:2229] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:19:02.017093 7550 raft_consensus.cc:2258] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:19:02.017318 7550 ts_tablet_manager.cc:1905] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:19:02.017568 7548 ts_tablet_manager.cc:1918] T 77a2fc42fe194514a812efc6770a20b0 P 82056c4aa10a4f63b50c264c3e62a0a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:19:02.017618 7548 log.cc:1196] T 77a2fc42fe194514a812efc6770a20b0 P 82056c4aa10a4f63b50c264c3e62a0a4: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/77a2fc42fe194514a812efc6770a20b0
I20260924 15:19:02.017839 7548 ts_tablet_manager.cc:1939] T 77a2fc42fe194514a812efc6770a20b0 P 82056c4aa10a4f63b50c264c3e62a0a4: Deleting consensus metadata
I20260924 15:19:02.018126 6952 catalog_manager.cc:4841] TS 82056c4aa10a4f63b50c264c3e62a0a4 (127.31.12.66:35055): tablet 77a2fc42fe194514a812efc6770a20b0 (table test-table [id=ff177e56e32d45ddb46f670e68e1142f]) successfully deleted
I20260924 15:19:02.018800 7550 ts_tablet_manager.cc:1918] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:19:02.018838 7550 log.cc:1196] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/77a2fc42fe194514a812efc6770a20b0
I20260924 15:19:02.019006 7550 ts_tablet_manager.cc:1939] T 77a2fc42fe194514a812efc6770a20b0 P c97581888e264d1fa8711d8c8b66736a: Deleting consensus metadata
I20260924 15:19:02.019191 7549 raft_consensus.cc:2229] T 77a2fc42fe194514a812efc6770a20b0 P 0510948451454cf994e41c6f5390beb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:19:02.019304 7549 raft_consensus.cc:2258] T 77a2fc42fe194514a812efc6770a20b0 P 0510948451454cf994e41c6f5390beb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:19:02.019345 6952 catalog_manager.cc:4841] TS c97581888e264d1fa8711d8c8b66736a (127.31.12.67:37441): tablet 77a2fc42fe194514a812efc6770a20b0 (table test-table [id=ff177e56e32d45ddb46f670e68e1142f]) successfully deleted
I20260924 15:19:02.019596 7549 ts_tablet_manager.cc:1905] T 77a2fc42fe194514a812efc6770a20b0 P 0510948451454cf994e41c6f5390beb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:19:02.020831 7549 ts_tablet_manager.cc:1918] T 77a2fc42fe194514a812efc6770a20b0 P 0510948451454cf994e41c6f5390beb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:19:02.020885 7549 log.cc:1196] T 77a2fc42fe194514a812efc6770a20b0 P 0510948451454cf994e41c6f5390beb0: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/wal/wals/77a2fc42fe194514a812efc6770a20b0
I20260924 15:19:02.021055 7549 ts_tablet_manager.cc:1939] T 77a2fc42fe194514a812efc6770a20b0 P 0510948451454cf994e41c6f5390beb0: Deleting consensus metadata
I20260924 15:19:02.021382 6953 catalog_manager.cc:4841] TS 0510948451454cf994e41c6f5390beb0 (127.31.12.65:34463): tablet 77a2fc42fe194514a812efc6770a20b0 (table test-table [id=ff177e56e32d45ddb46f670e68e1142f]) successfully deleted
Sep 24 15:19:02 dist-test-slave-zwbb krb5kdc[6921](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 15:19:02 dist-test-slave-zwbb krb5kdc[6921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:02 dist-test-slave-zwbb krb5kdc[6921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 15:19:02.042717 7522 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:02.036943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35032 (local address 127.31.12.126:43239)
0924 15:19:02.037037 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:02.037040 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:02.037141 (+ 101us) server_negotiation.cc:407] Connection header received
0924 15:19:02.037245 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:02.037246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:02.037260 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:02.037327 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:02.037942 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:02.038461 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:02.039198 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:02.039407 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:02.039488 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:02.042644 (+ 3156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35032: BlockingRecv error: recv got EOF from 127.0.0.1:35032 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 15:19:02 dist-test-slave-zwbb krb5kdc[6921](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM
I20260924 15:19:02.051858 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:02.043790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35048 (local address 127.31.12.126:43239)
0924 15:19:02.043829 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:02.043831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:02.043855 (+ 24us) server_negotiation.cc:407] Connection header received
0924 15:19:02.043930 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:02.043931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:02.043941 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:02.044007 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:02.044589 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:02.045096 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:02.046163 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:02.046406 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:02.046474 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:02.049023 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:02.049025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:02.049026 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:19:02.049033 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:02.050646 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:02.050989 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:02.050991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:02.050991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:02.051053 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:02.051247 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:02.051248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:02.051249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:02.051408 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:02.051534 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:02.051652 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 15:19:02.051747 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:19:02.053632 6965 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:19:02.053812 6965 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:19:02.055146 7104 tablet_service.cc:1467] Processing CreateTablet for tablet b451577e36d84943a87aebec265ac296 (DEFAULT_TABLE table=test-table [id=8fa94acf0180489cb475393122008ec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:19:02.055215 7245 tablet_service.cc:1467] Processing CreateTablet for tablet b451577e36d84943a87aebec265ac296 (DEFAULT_TABLE table=test-table [id=8fa94acf0180489cb475393122008ec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:19:02.055209 7386 tablet_service.cc:1467] Processing CreateTablet for tablet b451577e36d84943a87aebec265ac296 (DEFAULT_TABLE table=test-table [id=8fa94acf0180489cb475393122008ec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:19:02.055291 7104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b451577e36d84943a87aebec265ac296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:02.055336 7245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b451577e36d84943a87aebec265ac296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:02.055337 7386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b451577e36d84943a87aebec265ac296. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:02.056665 7482 tablet_bootstrap.cc:492] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a: Bootstrap starting.
I20260924 15:19:02.056703 7483 tablet_bootstrap.cc:492] T b451577e36d84943a87aebec265ac296 P 82056c4aa10a4f63b50c264c3e62a0a4: Bootstrap starting.
I20260924 15:19:02.057070 7482 tablet_bootstrap.cc:654] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:02.057097 7483 tablet_bootstrap.cc:654] T b451577e36d84943a87aebec265ac296 P 82056c4aa10a4f63b50c264c3e62a0a4: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:02.057760 7485 tablet_bootstrap.cc:492] T b451577e36d84943a87aebec265ac296 P 0510948451454cf994e41c6f5390beb0: Bootstrap starting.
I20260924 15:19:02.057864 7482 tablet_bootstrap.cc:492] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a: No bootstrap required, opened a new log
I20260924 15:19:02.057924 7482 ts_tablet_manager.cc:1397] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 15:19:02.058097 7485 tablet_bootstrap.cc:654] T b451577e36d84943a87aebec265ac296 P 0510948451454cf994e41c6f5390beb0: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:02.058091 7482 raft_consensus.cc:348] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:02.058173 7482 raft_consensus.cc:374] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:02.058198 7482 raft_consensus.cc:729] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97581888e264d1fa8711d8c8b66736a, State: Initialized, Role: FOLLOWER
I20260924 15:19:02.058243 7482 consensus_queue.cc:260] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:02.058336 7482 ts_tablet_manager.cc:1428] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:02.058605 7483 tablet_bootstrap.cc:492] T b451577e36d84943a87aebec265ac296 P 82056c4aa10a4f63b50c264c3e62a0a4: No bootstrap required, opened a new log
I20260924 15:19:02.058657 7483 ts_tablet_manager.cc:1397] T b451577e36d84943a87aebec265ac296 P 82056c4aa10a4f63b50c264c3e62a0a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:19:02.058796 7483 raft_consensus.cc:348] T b451577e36d84943a87aebec265ac296 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:02.058859 7483 raft_consensus.cc:374] T b451577e36d84943a87aebec265ac296 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:02.058877 7483 raft_consensus.cc:729] T b451577e36d84943a87aebec265ac296 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82056c4aa10a4f63b50c264c3e62a0a4, State: Initialized, Role: FOLLOWER
I20260924 15:19:02.058921 7483 consensus_queue.cc:260] T b451577e36d84943a87aebec265ac296 P 82056c4aa10a4f63b50c264c3e62a0a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:02.059007 7483 ts_tablet_manager.cc:1428] T b451577e36d84943a87aebec265ac296 P 82056c4aa10a4f63b50c264c3e62a0a4: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:02.059466 7485 tablet_bootstrap.cc:492] T b451577e36d84943a87aebec265ac296 P 0510948451454cf994e41c6f5390beb0: No bootstrap required, opened a new log
I20260924 15:19:02.059521 7485 ts_tablet_manager.cc:1397] T b451577e36d84943a87aebec265ac296 P 0510948451454cf994e41c6f5390beb0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260924 15:19:02.059643 7485 raft_consensus.cc:348] T b451577e36d84943a87aebec265ac296 P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:02.059719 7485 raft_consensus.cc:374] T b451577e36d84943a87aebec265ac296 P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:02.059746 7485 raft_consensus.cc:729] T b451577e36d84943a87aebec265ac296 P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0510948451454cf994e41c6f5390beb0, State: Initialized, Role: FOLLOWER
I20260924 15:19:02.059794 7485 consensus_queue.cc:260] T b451577e36d84943a87aebec265ac296 P 0510948451454cf994e41c6f5390beb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:02.059900 7485 ts_tablet_manager.cc:1428] T b451577e36d84943a87aebec265ac296 P 0510948451454cf994e41c6f5390beb0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:02.165064 7514 raft_consensus.cc:482] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:19:02.165164 7514 raft_consensus.cc:504] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:02.165323 7514 leader_election.cc:288] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0510948451454cf994e41c6f5390beb0 (127.31.12.65:34463), 82056c4aa10a4f63b50c264c3e62a0a4 (127.31.12.66:35055)
I20260924 15:19:02.165575 7265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b451577e36d84943a87aebec265ac296" candidate_uuid: "c97581888e264d1fa8711d8c8b66736a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" is_pre_election: true
I20260924 15:19:02.165584 7124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b451577e36d84943a87aebec265ac296" candidate_uuid: "c97581888e264d1fa8711d8c8b66736a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0510948451454cf994e41c6f5390beb0" is_pre_election: true
I20260924 15:19:02.165679 7265 raft_consensus.cc:2454] T b451577e36d84943a87aebec265ac296 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c97581888e264d1fa8711d8c8b66736a in term 0.
I20260924 15:19:02.165679 7124 raft_consensus.cc:2454] T b451577e36d84943a87aebec265ac296 P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c97581888e264d1fa8711d8c8b66736a in term 0.
I20260924 15:19:02.165841 7338 leader_election.cc:302] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82056c4aa10a4f63b50c264c3e62a0a4, c97581888e264d1fa8711d8c8b66736a; no voters:
I20260924 15:19:02.165930 7514 raft_consensus.cc:2781] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:19:02.165969 7514 raft_consensus.cc:482] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:19:02.165988 7514 raft_consensus.cc:3037] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:02.166692 7514 raft_consensus.cc:504] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:02.166832 7514 leader_election.cc:288] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [CANDIDATE]: Term 1 election: Requested vote from peers 0510948451454cf994e41c6f5390beb0 (127.31.12.65:34463), 82056c4aa10a4f63b50c264c3e62a0a4 (127.31.12.66:35055)
I20260924 15:19:02.166967 7124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b451577e36d84943a87aebec265ac296" candidate_uuid: "c97581888e264d1fa8711d8c8b66736a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0510948451454cf994e41c6f5390beb0"
I20260924 15:19:02.167070 7124 raft_consensus.cc:3037] T b451577e36d84943a87aebec265ac296 P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:02.167021 7265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b451577e36d84943a87aebec265ac296" candidate_uuid: "c97581888e264d1fa8711d8c8b66736a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82056c4aa10a4f63b50c264c3e62a0a4"
I20260924 15:19:02.167115 7265 raft_consensus.cc:3037] T b451577e36d84943a87aebec265ac296 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:02.167691 7124 raft_consensus.cc:2454] T b451577e36d84943a87aebec265ac296 P 0510948451454cf994e41c6f5390beb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c97581888e264d1fa8711d8c8b66736a in term 1.
I20260924 15:19:02.167862 7339 leader_election.cc:302] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0510948451454cf994e41c6f5390beb0, c97581888e264d1fa8711d8c8b66736a; no voters:
I20260924 15:19:02.167941 7265 raft_consensus.cc:2454] T b451577e36d84943a87aebec265ac296 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c97581888e264d1fa8711d8c8b66736a in term 1.
I20260924 15:19:02.167964 7514 raft_consensus.cc:2781] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:19:02.168011 7514 raft_consensus.cc:686] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [term 1 LEADER]: Becoming Leader. State: Replica: c97581888e264d1fa8711d8c8b66736a, State: Running, Role: LEADER
I20260924 15:19:02.168066 7514 consensus_queue.cc:237] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:02.168749 6966 catalog_manager.cc:5495] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a reported cstate change: term changed from 0 to 1, leader changed from <none> to c97581888e264d1fa8711d8c8b66736a (127.31.12.67). New cstate: current_term: 1 leader_uuid: "c97581888e264d1fa8711d8c8b66736a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } health_report { overall_health: HEALTHY } } }
I20260924 15:19:02.178815 7530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:02.176612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.31.12.67:37441)
0924 15:19:02.176676 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:02.176679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:02.176690 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:19:02.176716 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:02.176717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:02.176727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:02.176760 (+ 33us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:19:02.176930 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:02.177386 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:02.178143 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:02.178333 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:02.178405 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:02.178566 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:19:02.178598 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:19:02.178655 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:02.178716 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 15:19:02.178734 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:19:02.179585 7124 raft_consensus.cc:1261] T b451577e36d84943a87aebec265ac296 P 0510948451454cf994e41c6f5390beb0 [term 1 FOLLOWER]: Refusing update from remote peer c97581888e264d1fa8711d8c8b66736a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:19:02.179606 7265 raft_consensus.cc:1261] T b451577e36d84943a87aebec265ac296 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 FOLLOWER]: Refusing update from remote peer c97581888e264d1fa8711d8c8b66736a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:19:02.179800 7546 consensus_queue.cc:1035] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:19:02.179920 7546 consensus_queue.cc:1035] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [LEADER]: Connected to new peer: Peer: permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:19:02.182778 6966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:19:02.182868 6966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:19:02.183509 6966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974: Sending DeleteTablet for 3 replicas of tablet b451577e36d84943a87aebec265ac296
I20260924 15:19:02.183744 7245 tablet_service.cc:1514] Processing DeleteTablet for tablet b451577e36d84943a87aebec265ac296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:19:02 UTC) from {username='oryx', principal='oryx/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:33884
I20260924 15:19:02.183779 7104 tablet_service.cc:1514] Processing DeleteTablet for tablet b451577e36d84943a87aebec265ac296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:19:02 UTC) from {username='oryx', principal='oryx/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:46380
I20260924 15:19:02.183773 7386 tablet_service.cc:1514] Processing DeleteTablet for tablet b451577e36d84943a87aebec265ac296 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:19:02 UTC) from {username='oryx', principal='oryx/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:53908
I20260924 15:19:02.183861 7550 tablet_replica.cc:330] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a: stopping tablet replica
I20260924 15:19:02.183863 7548 tablet_replica.cc:330] T b451577e36d84943a87aebec265ac296 P 82056c4aa10a4f63b50c264c3e62a0a4: stopping tablet replica
I20260924 15:19:02.183908 7550 raft_consensus.cc:2229] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:19:02.183914 7548 raft_consensus.cc:2229] T b451577e36d84943a87aebec265ac296 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:19:02.183952 7548 raft_consensus.cc:2258] T b451577e36d84943a87aebec265ac296 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:19:02.183960 7550 raft_consensus.cc:2258] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:19:02.184026 7549 tablet_replica.cc:330] T b451577e36d84943a87aebec265ac296 P 0510948451454cf994e41c6f5390beb0: stopping tablet replica
I20260924 15:19:02.184077 7549 raft_consensus.cc:2229] T b451577e36d84943a87aebec265ac296 P 0510948451454cf994e41c6f5390beb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:19:02.184124 7549 raft_consensus.cc:2258] T b451577e36d84943a87aebec265ac296 P 0510948451454cf994e41c6f5390beb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:19:02.184401 7550 ts_tablet_manager.cc:1905] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:19:02.184460 7549 ts_tablet_manager.cc:1905] T b451577e36d84943a87aebec265ac296 P 0510948451454cf994e41c6f5390beb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:19:02.184607 6966 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:19:02.184738 6966 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:19:02.186045 7550 ts_tablet_manager.cc:1918] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:19:02.186075 7549 ts_tablet_manager.cc:1918] T b451577e36d84943a87aebec265ac296 P 0510948451454cf994e41c6f5390beb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:19:02.186100 7550 log.cc:1196] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/b451577e36d84943a87aebec265ac296
I20260924 15:19:02.186128 7549 log.cc:1196] T b451577e36d84943a87aebec265ac296 P 0510948451454cf994e41c6f5390beb0: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/wal/wals/b451577e36d84943a87aebec265ac296
I20260924 15:19:02.186333 7550 ts_tablet_manager.cc:1939] T b451577e36d84943a87aebec265ac296 P c97581888e264d1fa8711d8c8b66736a: Deleting consensus metadata
I20260924 15:19:02.186349 7386 tablet_service.cc:1467] Processing CreateTablet for tablet c8edacfebf7f4fd28bd4e6bc52168622 (DEFAULT_TABLE table=test-table [id=14873a83b6524ad79a171a6f883cc16f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:19:02.186369 7549 ts_tablet_manager.cc:1939] T b451577e36d84943a87aebec265ac296 P 0510948451454cf994e41c6f5390beb0: Deleting consensus metadata
I20260924 15:19:02.186436 7386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8edacfebf7f4fd28bd4e6bc52168622. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:02.186131 7104 tablet_service.cc:1467] Processing CreateTablet for tablet c8edacfebf7f4fd28bd4e6bc52168622 (DEFAULT_TABLE table=test-table [id=14873a83b6524ad79a171a6f883cc16f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:19:02.186579 7104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8edacfebf7f4fd28bd4e6bc52168622. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:02.187330 7482 tablet_bootstrap.cc:492] T c8edacfebf7f4fd28bd4e6bc52168622 P c97581888e264d1fa8711d8c8b66736a: Bootstrap starting.
I20260924 15:19:02.187654 7482 tablet_bootstrap.cc:654] T c8edacfebf7f4fd28bd4e6bc52168622 P c97581888e264d1fa8711d8c8b66736a: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:02.187723 7485 tablet_bootstrap.cc:492] T c8edacfebf7f4fd28bd4e6bc52168622 P 0510948451454cf994e41c6f5390beb0: Bootstrap starting.
I20260924 15:19:02.187876 6952 catalog_manager.cc:4841] TS c97581888e264d1fa8711d8c8b66736a (127.31.12.67:37441): tablet b451577e36d84943a87aebec265ac296 (table test-table [id=8fa94acf0180489cb475393122008ec2]) successfully deleted
I20260924 15:19:02.187990 7485 tablet_bootstrap.cc:654] T c8edacfebf7f4fd28bd4e6bc52168622 P 0510948451454cf994e41c6f5390beb0: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:02.188126 7548 ts_tablet_manager.cc:1905] T b451577e36d84943a87aebec265ac296 P 82056c4aa10a4f63b50c264c3e62a0a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:19:02.188239 7245 tablet_service.cc:1467] Processing CreateTablet for tablet c8edacfebf7f4fd28bd4e6bc52168622 (DEFAULT_TABLE table=test-table [id=14873a83b6524ad79a171a6f883cc16f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:19:02.188340 7245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8edacfebf7f4fd28bd4e6bc52168622. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:02.188478 7485 tablet_bootstrap.cc:492] T c8edacfebf7f4fd28bd4e6bc52168622 P 0510948451454cf994e41c6f5390beb0: No bootstrap required, opened a new log
I20260924 15:19:02.188530 7485 ts_tablet_manager.cc:1397] T c8edacfebf7f4fd28bd4e6bc52168622 P 0510948451454cf994e41c6f5390beb0: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 15:19:02.188623 7485 raft_consensus.cc:348] T c8edacfebf7f4fd28bd4e6bc52168622 P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:02.188678 7485 raft_consensus.cc:374] T c8edacfebf7f4fd28bd4e6bc52168622 P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:02.188696 7485 raft_consensus.cc:729] T c8edacfebf7f4fd28bd4e6bc52168622 P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0510948451454cf994e41c6f5390beb0, State: Initialized, Role: FOLLOWER
I20260924 15:19:02.188736 7485 consensus_queue.cc:260] T c8edacfebf7f4fd28bd4e6bc52168622 P 0510948451454cf994e41c6f5390beb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:02.188824 7485 ts_tablet_manager.cc:1428] T c8edacfebf7f4fd28bd4e6bc52168622 P 0510948451454cf994e41c6f5390beb0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:02.189747 7548 ts_tablet_manager.cc:1918] T b451577e36d84943a87aebec265ac296 P 82056c4aa10a4f63b50c264c3e62a0a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:19:02.189790 7548 log.cc:1196] T b451577e36d84943a87aebec265ac296 P 82056c4aa10a4f63b50c264c3e62a0a4: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/b451577e36d84943a87aebec265ac296
I20260924 15:19:02.189889 7483 tablet_bootstrap.cc:492] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4: Bootstrap starting.
I20260924 15:19:02.190027 7548 ts_tablet_manager.cc:1939] T b451577e36d84943a87aebec265ac296 P 82056c4aa10a4f63b50c264c3e62a0a4: Deleting consensus metadata
I20260924 15:19:02.190284 7483 tablet_bootstrap.cc:654] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:02.190459 6953 catalog_manager.cc:4841] TS 0510948451454cf994e41c6f5390beb0 (127.31.12.65:34463): tablet b451577e36d84943a87aebec265ac296 (table test-table [id=8fa94acf0180489cb475393122008ec2]) successfully deleted
I20260924 15:19:02.190789 6952 catalog_manager.cc:4841] TS 82056c4aa10a4f63b50c264c3e62a0a4 (127.31.12.66:35055): tablet b451577e36d84943a87aebec265ac296 (table test-table [id=8fa94acf0180489cb475393122008ec2]) successfully deleted
I20260924 15:19:02.190899 7482 tablet_bootstrap.cc:492] T c8edacfebf7f4fd28bd4e6bc52168622 P c97581888e264d1fa8711d8c8b66736a: No bootstrap required, opened a new log
I20260924 15:19:02.190945 7482 ts_tablet_manager.cc:1397] T c8edacfebf7f4fd28bd4e6bc52168622 P c97581888e264d1fa8711d8c8b66736a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.000s
I20260924 15:19:02.191056 7482 raft_consensus.cc:348] T c8edacfebf7f4fd28bd4e6bc52168622 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:02.191134 7482 raft_consensus.cc:374] T c8edacfebf7f4fd28bd4e6bc52168622 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:02.191165 7482 raft_consensus.cc:729] T c8edacfebf7f4fd28bd4e6bc52168622 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c97581888e264d1fa8711d8c8b66736a, State: Initialized, Role: FOLLOWER
I20260924 15:19:02.191206 7482 consensus_queue.cc:260] T c8edacfebf7f4fd28bd4e6bc52168622 P c97581888e264d1fa8711d8c8b66736a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:02.191295 7482 ts_tablet_manager.cc:1428] T c8edacfebf7f4fd28bd4e6bc52168622 P c97581888e264d1fa8711d8c8b66736a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:02.191533 7483 tablet_bootstrap.cc:492] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4: No bootstrap required, opened a new log
I20260924 15:19:02.191594 7483 ts_tablet_manager.cc:1397] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:19:02.191740 7483 raft_consensus.cc:348] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:02.191808 7483 raft_consensus.cc:374] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:02.191835 7483 raft_consensus.cc:729] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82056c4aa10a4f63b50c264c3e62a0a4, State: Initialized, Role: FOLLOWER
I20260924 15:19:02.191879 7483 consensus_queue.cc:260] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:02.191967 7483 ts_tablet_manager.cc:1428] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 15:19:02.247145 7490 raft_consensus.cc:482] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:19:02.247233 7490 raft_consensus.cc:504] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:02.247412 7490 leader_election.cc:288] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0510948451454cf994e41c6f5390beb0 (127.31.12.65:34463), c97581888e264d1fa8711d8c8b66736a (127.31.12.67:37441)
I20260924 15:19:02.247609 7406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8edacfebf7f4fd28bd4e6bc52168622" candidate_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97581888e264d1fa8711d8c8b66736a" is_pre_election: true
I20260924 15:19:02.247605 7124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8edacfebf7f4fd28bd4e6bc52168622" candidate_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0510948451454cf994e41c6f5390beb0" is_pre_election: true
I20260924 15:19:02.247712 7124 raft_consensus.cc:2454] T c8edacfebf7f4fd28bd4e6bc52168622 P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82056c4aa10a4f63b50c264c3e62a0a4 in term 0.
I20260924 15:19:02.247713 7406 raft_consensus.cc:2454] T c8edacfebf7f4fd28bd4e6bc52168622 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82056c4aa10a4f63b50c264c3e62a0a4 in term 0.
I20260924 15:19:02.247859 7199 leader_election.cc:302] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0510948451454cf994e41c6f5390beb0, 82056c4aa10a4f63b50c264c3e62a0a4; no voters:
I20260924 15:19:02.247972 7490 raft_consensus.cc:2781] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:19:02.248020 7490 raft_consensus.cc:482] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:19:02.248040 7490 raft_consensus.cc:3037] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:02.248874 7490 raft_consensus.cc:504] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:02.249011 7490 leader_election.cc:288] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [CANDIDATE]: Term 1 election: Requested vote from peers 0510948451454cf994e41c6f5390beb0 (127.31.12.65:34463), c97581888e264d1fa8711d8c8b66736a (127.31.12.67:37441)
I20260924 15:19:02.249159 7406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8edacfebf7f4fd28bd4e6bc52168622" candidate_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c97581888e264d1fa8711d8c8b66736a"
I20260924 15:19:02.249166 7124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8edacfebf7f4fd28bd4e6bc52168622" candidate_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0510948451454cf994e41c6f5390beb0"
I20260924 15:19:02.249254 7124 raft_consensus.cc:3037] T c8edacfebf7f4fd28bd4e6bc52168622 P 0510948451454cf994e41c6f5390beb0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:02.249254 7406 raft_consensus.cc:3037] T c8edacfebf7f4fd28bd4e6bc52168622 P c97581888e264d1fa8711d8c8b66736a [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:02.249893 7406 raft_consensus.cc:2454] T c8edacfebf7f4fd28bd4e6bc52168622 P c97581888e264d1fa8711d8c8b66736a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82056c4aa10a4f63b50c264c3e62a0a4 in term 1.
I20260924 15:19:02.250030 7124 raft_consensus.cc:2454] T c8edacfebf7f4fd28bd4e6bc52168622 P 0510948451454cf994e41c6f5390beb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82056c4aa10a4f63b50c264c3e62a0a4 in term 1.
I20260924 15:19:02.250084 7198 leader_election.cc:302] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82056c4aa10a4f63b50c264c3e62a0a4, c97581888e264d1fa8711d8c8b66736a; no voters:
I20260924 15:19:02.250200 7490 raft_consensus.cc:2781] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:19:02.250348 7490 raft_consensus.cc:686] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 LEADER]: Becoming Leader. State: Replica: 82056c4aa10a4f63b50c264c3e62a0a4, State: Running, Role: LEADER
I20260924 15:19:02.250427 7490 consensus_queue.cc:237] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } }
I20260924 15:19:02.251103 6965 catalog_manager.cc:5495] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82056c4aa10a4f63b50c264c3e62a0a4 (127.31.12.66). New cstate: current_term: 1 leader_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82056c4aa10a4f63b50c264c3e62a0a4" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 } health_report { overall_health: UNKNOWN } } }
I20260924 15:19:02.266862 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:02.264285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33890 (local address 127.31.12.66:35055)
0924 15:19:02.264442 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:02.264444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:02.264450 (+ 6us) server_negotiation.cc:407] Connection header received
0924 15:19:02.264485 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:02.264485 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:02.264503 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:02.264598 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:19:02.264704 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:02.265262 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:02.266048 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:02.266271 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:02.266351 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:02.266512 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:19:02.266626 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:19:02.266708 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:02.266747 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 15:19:02.266768 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260924 15:19:02.271497 7530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:02.269262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:35519 (local address 127.31.12.67:37441)
0924 15:19:02.269332 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:02.269333 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:02.269353 (+ 20us) server_negotiation.cc:407] Connection header received
0924 15:19:02.269416 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:02.269417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:02.269424 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:02.269521 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:19:02.269641 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:02.270098 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:02.270911 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:02.271088 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:02.271145 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:02.271307 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:19:02.271330 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:19:02.271374 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:02.271428 (+ 54us) server_negotiation.cc:299] Negotiation successful
0924 15:19:02.271445 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 15:19:02.271520 7578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:02.269120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:37441 (local address 127.31.12.66:35519)
0924 15:19:02.269300 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:02.269311 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:02.269366 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:02.269525 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:02.269526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:02.269528 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:19:02.269583 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:02.269584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:02.270204 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:02.270205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:02.270826 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:02.270832 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:02.271169 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:02.271170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:02.271213 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:02.271225 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:19:02.271386 (+ 161us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:19:02.271388 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:19:02.271422 (+ 34us) client_negotiation.cc:240] Negotiation successful
0924 15:19:02.271454 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260924 15:19:02.274102 7406 raft_consensus.cc:1261] T c8edacfebf7f4fd28bd4e6bc52168622 P c97581888e264d1fa8711d8c8b66736a [term 1 FOLLOWER]: Refusing update from remote peer 82056c4aa10a4f63b50c264c3e62a0a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:19:02.274139 7124 raft_consensus.cc:1261] T c8edacfebf7f4fd28bd4e6bc52168622 P 0510948451454cf994e41c6f5390beb0 [term 1 FOLLOWER]: Refusing update from remote peer 82056c4aa10a4f63b50c264c3e62a0a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:19:02.274456 7573 consensus_queue.cc:1035] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c97581888e264d1fa8711d8c8b66736a" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 37441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:19:02.274601 7490 consensus_queue.cc:1035] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0510948451454cf994e41c6f5390beb0" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:19:02.281724 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:02.279172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:41387 (local address 127.31.12.66:35055)
0924 15:19:02.279209 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:02.279211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:02.279241 (+ 30us) server_negotiation.cc:407] Connection header received
0924 15:19:02.279299 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:02.279300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:02.279307 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:02.279378 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:19:02.279543 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:02.280202 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:02.280870 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:02.281199 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:02.281274 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:02.281476 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:19:02.281513 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:19:02.281566 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:02.281628 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 15:19:02.281652 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:19:02.281762 7543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:02.279139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:35055 (local address 127.31.12.67:41387)
0924 15:19:02.279195 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:02.279206 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:02.279254 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:02.279441 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:02.279443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:02.279444 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 15:19:02.279486 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:02.279488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:02.280263 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:02.280263 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:02.280805 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:02.280807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:02.281310 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:02.281311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:02.281372 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:02.281377 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 15:19:02.281574 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 15:19:02.281576 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:19:02.281625 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 15:19:02.281681 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 15:19:02.288363 6965 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:19:02.288465 6965 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:19:02.289064 6965 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6d9849a34d749fb9d585de3ff9f7974: Sending DeleteTablet for 3 replicas of tablet c8edacfebf7f4fd28bd4e6bc52168622
I20260924 15:19:02.289292 7245 tablet_service.cc:1514] Processing DeleteTablet for tablet c8edacfebf7f4fd28bd4e6bc52168622 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:19:02 UTC) from {username='oryx', principal='oryx/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:33884
I20260924 15:19:02.289321 7104 tablet_service.cc:1514] Processing DeleteTablet for tablet c8edacfebf7f4fd28bd4e6bc52168622 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:19:02 UTC) from {username='oryx', principal='oryx/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:46380
I20260924 15:19:02.289320 7386 tablet_service.cc:1514] Processing DeleteTablet for tablet c8edacfebf7f4fd28bd4e6bc52168622 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:19:02 UTC) from {username='oryx', principal='oryx/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:53908
I20260924 15:19:02.289381 7548 tablet_replica.cc:330] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4: stopping tablet replica
I20260924 15:19:02.289405 7550 tablet_replica.cc:330] T c8edacfebf7f4fd28bd4e6bc52168622 P c97581888e264d1fa8711d8c8b66736a: stopping tablet replica
I20260924 15:19:02.289413 7549 tablet_replica.cc:330] T c8edacfebf7f4fd28bd4e6bc52168622 P 0510948451454cf994e41c6f5390beb0: stopping tablet replica
I20260924 15:19:02.289429 7548 raft_consensus.cc:2229] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:19:02.289484 7550 raft_consensus.cc:2229] T c8edacfebf7f4fd28bd4e6bc52168622 P c97581888e264d1fa8711d8c8b66736a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:19:02.289487 7549 raft_consensus.cc:2229] T c8edacfebf7f4fd28bd4e6bc52168622 P 0510948451454cf994e41c6f5390beb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:19:02.289516 7548 raft_consensus.cc:2258] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:19:02.289520 7549 raft_consensus.cc:2258] T c8edacfebf7f4fd28bd4e6bc52168622 P 0510948451454cf994e41c6f5390beb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:19:02.289520 7550 raft_consensus.cc:2258] T c8edacfebf7f4fd28bd4e6bc52168622 P c97581888e264d1fa8711d8c8b66736a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:19:02.289898 7549 ts_tablet_manager.cc:1905] T c8edacfebf7f4fd28bd4e6bc52168622 P 0510948451454cf994e41c6f5390beb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:19:02.289906 7548 ts_tablet_manager.cc:1905] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:19:02.291612 7550 ts_tablet_manager.cc:1905] T c8edacfebf7f4fd28bd4e6bc52168622 P c97581888e264d1fa8711d8c8b66736a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:19:02.291625 7549 ts_tablet_manager.cc:1918] T c8edacfebf7f4fd28bd4e6bc52168622 P 0510948451454cf994e41c6f5390beb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:19:02.291671 7549 log.cc:1196] T c8edacfebf7f4fd28bd4e6bc52168622 P 0510948451454cf994e41c6f5390beb0: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-0/wal/wals/c8edacfebf7f4fd28bd4e6bc52168622
I20260924 15:19:02.291834 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 7039
I20260924 15:19:02.291879 7549 ts_tablet_manager.cc:1939] T c8edacfebf7f4fd28bd4e6bc52168622 P 0510948451454cf994e41c6f5390beb0: Deleting consensus metadata
I20260924 15:19:02.293326 7550 ts_tablet_manager.cc:1918] T c8edacfebf7f4fd28bd4e6bc52168622 P c97581888e264d1fa8711d8c8b66736a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:19:02.293387 7550 log.cc:1196] T c8edacfebf7f4fd28bd4e6bc52168622 P c97581888e264d1fa8711d8c8b66736a: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/c8edacfebf7f4fd28bd4e6bc52168622
I20260924 15:19:02.293625 7550 ts_tablet_manager.cc:1939] T c8edacfebf7f4fd28bd4e6bc52168622 P c97581888e264d1fa8711d8c8b66736a: Deleting consensus metadata
I20260924 15:19:02.293833 7548 ts_tablet_manager.cc:1918] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 15:19:02.293886 7548 log.cc:1196] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/c8edacfebf7f4fd28bd4e6bc52168622
I20260924 15:19:02.293994 6952 catalog_manager.cc:4841] TS c97581888e264d1fa8711d8c8b66736a (127.31.12.67:37441): tablet c8edacfebf7f4fd28bd4e6bc52168622 (table test-table [id=14873a83b6524ad79a171a6f883cc16f]) successfully deleted
I20260924 15:19:02.294100 7548 ts_tablet_manager.cc:1939] T c8edacfebf7f4fd28bd4e6bc52168622 P 82056c4aa10a4f63b50c264c3e62a0a4: Deleting consensus metadata
I20260924 15:19:02.294353 6952 catalog_manager.cc:4841] TS 82056c4aa10a4f63b50c264c3e62a0a4 (127.31.12.66:35055): tablet c8edacfebf7f4fd28bd4e6bc52168622 (table test-table [id=14873a83b6524ad79a171a6f883cc16f]) successfully deleted
W20260924 15:19:02.300505 6953 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.12.65:34463 (error 108)
I20260924 15:19:02.300709 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 7180
I20260924 15:19:02.303014 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:02.302760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:34463 (local address 127.0.0.1:46396)
0924 15:19:02.302908 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:02.302962 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.65:34463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
W20260924 15:19:02.303155 6953 catalog_manager.cc:4584] TS 0510948451454cf994e41c6f5390beb0 (127.31.12.65:34463): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8edacfebf7f4fd28bd4e6bc52168622: Network error: Client connection negotiation failed: client connection to 127.31.12.65:34463: connect: Connection refused (error 111)
I20260924 15:19:02.306506 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 7320
I20260924 15:19:02.312426 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 6934
2026-09-24T15:19:02Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2881 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:19:02 dist-test-slave-zwbb krb5kdc[7587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:19:02 dist-test-slave-zwbb krb5kdc[7587](info): set up 2 sockets
Sep 24 15:19:02 dist-test-slave-zwbb krb5kdc[7587](info): commencing operation
krb5kdc: starting...
W20260924 15:19:03.371531 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:19:03 dist-test-slave-zwbb krb5kdc[7587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263143, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:19:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:19:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:19:03.486119 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:45283
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:40265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:45283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:19:03.570160 7601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:03.570379 7601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:03.570413 7601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:03.571530 7601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:19:03.571591 7601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:03.571615 7601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:03.571635 7601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:19:03.571663 7601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:19:03.573104 7601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:40265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:45283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:45283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:03.573364 7601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:03.573599 7601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:03.578168 7606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:03.578284 7609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:03.578382 7601 server_base.cc:1034] running on GCE node
W20260924 15:19:03.578168 7607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:03.578665 7601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:03.578939 7601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:03.580104 7601 hybrid_clock.cc:648] HybridClock initialized: now 1790263143580099 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T15:19:03Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1269 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:19:03 dist-test-slave-zwbb krb5kdc[7614](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:19:03 dist-test-slave-zwbb krb5kdc[7614](info): set up 2 sockets
Sep 24 15:19:03 dist-test-slave-zwbb krb5kdc[7614](info): commencing operation
krb5kdc: starting...
W20260924 15:19:04.629185 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:19:04 dist-test-slave-zwbb krb5kdc[7614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:19:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:19:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:19:04.749938 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:43993
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:39943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:43993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:19:04.832672 7627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:04.832913 7627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:04.832962 7627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:04.834144 7627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:19:04.834209 7627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:04.834235 7627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:04.834257 7627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:19:04.834285 7627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:19:04.835770 7627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:39943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:43993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:43993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:04.836046 7627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:04.836272 7627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:04.841101 7632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:04.841152 7635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:04.841178 7633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:04.841187 7627 server_base.cc:1034] running on GCE node
I20260924 15:19:04.841631 7627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:04.841943 7627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:04.843103 7627 hybrid_clock.cc:648] HybridClock initialized: now 1790263144843090 us; error 27 us; skew 500 ppm
Sep 24 15:19:04 dist-test-slave-zwbb krb5kdc[7614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263144, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:19:04.845616 7627 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:19:04.846082 7627 webserver.cc:460] Webserver started at http://127.31.12.126:35375/ using document root <none> and password file <none>
I20260924 15:19:04.846315 7627 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:04.846380 7627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:04.846513 7627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:19:04.847537 7627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "e8612f6c0a314205859f4ecb8997736b"
format_stamp: "Formatted at 2026-09-24 15:19:04 on dist-test-slave-zwbb"
server_key: "8f633462a6f79bb15e6bb660c419fc53"
server_key_iv: "eda57ada005de624eaebd14a66d0132b"
server_key_version: "encryptionkey@0"
I20260924 15:19:04.847893 7627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "e8612f6c0a314205859f4ecb8997736b"
format_stamp: "Formatted at 2026-09-24 15:19:04 on dist-test-slave-zwbb"
server_key: "8f633462a6f79bb15e6bb660c419fc53"
server_key_iv: "eda57ada005de624eaebd14a66d0132b"
server_key_version: "encryptionkey@0"
I20260924 15:19:04.849224 7627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:19:04.850247 7642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:04.850497 7627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:19:04.850610 7627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "e8612f6c0a314205859f4ecb8997736b"
format_stamp: "Formatted at 2026-09-24 15:19:04 on dist-test-slave-zwbb"
server_key: "8f633462a6f79bb15e6bb660c419fc53"
server_key_iv: "eda57ada005de624eaebd14a66d0132b"
server_key_version: "encryptionkey@0"
I20260924 15:19:04.850683 7627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:04.864802 7627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:04.866961 7627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:04.867084 7627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:04.870908 7627 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:43993
I20260924 15:19:04.870962 7695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:43993 every 8 connection(s)
I20260924 15:19:04.871343 7627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:19:04.872260 7696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:04.874693 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 7627
I20260924 15:19:04.874734 7696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b: Bootstrap starting.
I20260924 15:19:04.874812 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:19:04.875005 31793 external_mini_cluster.cc:1428] Setting key a5491e488cddb19b74419c4aee33d679
I20260924 15:19:04.875577 7696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:04.875908 7696 log.cc:824] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b: Log is configured to *not* fsync() on all Append() calls
I20260924 15:19:04.876616 7696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b: No bootstrap required, opened a new log
I20260924 15:19:04.878682 7696 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8612f6c0a314205859f4ecb8997736b" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43993 } }
I20260924 15:19:04.878844 7696 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:04.878924 7696 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8612f6c0a314205859f4ecb8997736b, State: Initialized, Role: FOLLOWER
I20260924 15:19:04.879101 7696 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8612f6c0a314205859f4ecb8997736b" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43993 } }
I20260924 15:19:04.879204 7696 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:19:04.879252 7696 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:19:04.879307 7696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:04.879877 7696 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8612f6c0a314205859f4ecb8997736b" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43993 } }
I20260924 15:19:04.880017 7696 leader_election.cc:302] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8612f6c0a314205859f4ecb8997736b; no voters:
I20260924 15:19:04.880234 7696 leader_election.cc:288] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:19:04.880260 7701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:19:04.880396 7701 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [term 1 LEADER]: Becoming Leader. State: Replica: e8612f6c0a314205859f4ecb8997736b, State: Running, Role: LEADER
I20260924 15:19:04.880515 7696 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:19:04.880493 7701 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8612f6c0a314205859f4ecb8997736b" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43993 } }
I20260924 15:19:04.880841 7701 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8612f6c0a314205859f4ecb8997736b. Latest consensus state: current_term: 1 leader_uuid: "e8612f6c0a314205859f4ecb8997736b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8612f6c0a314205859f4ecb8997736b" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43993 } } }
I20260924 15:19:04.880939 7701 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [sys.catalog]: This master's current role is: LEADER
I20260924 15:19:04.880856 7703 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8612f6c0a314205859f4ecb8997736b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8612f6c0a314205859f4ecb8997736b" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43993 } } }
I20260924 15:19:04.881114 7703 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [sys.catalog]: This master's current role is: LEADER
I20260924 15:19:04.881174 7706 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 15:19:04 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:19:04.882949 7706 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:19:04.884414 7706 catalog_manager.cc:1347] Generated new cluster ID: 1c1b5f75607747da889f112238ff8827
I20260924 15:19:04.884464 7706 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:19:04.886184 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:04.875509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33896 (local address 127.31.12.126:43993)
0924 15:19:04.875753 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:04.875762 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:04.875791 (+ 29us) server_negotiation.cc:407] Connection header received
0924 15:19:04.876280 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:04.876287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:04.876333 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:04.876476 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:04.877118 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:04.878054 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:04.878796 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:04.879355 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:04.879457 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:04.882277 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:04.882281 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:04.882282 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:19:04.882294 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:04.884501 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:04.884889 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:04.884892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:04.884893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:04.885008 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:04.885268 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:04.885270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:04.885270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:04.885620 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:04.885777 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:04.885852 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 15:19:04.885994 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":88,"threads_started":1}
I20260924 15:19:04.907359 7706 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:19:04.907898 7706 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:19:04.916388 7706 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b: Generated new TSK 0
I20260924 15:19:04.916591 7706 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
I20260924 15:19:04.948083 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:43993
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:39943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:19:05.029186 7722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:05.029387 7722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:05.029412 7722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:05.030593 7722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:05.030639 7722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:05.030666 7722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:19:05.032114 7722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:39943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:05.032294 7722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:05.032470 7722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:05.037381 7730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:05.037504 7722 server_base.cc:1034] running on GCE node
W20260924 15:19:05.037405 7727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:05.037657 7728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:05.037880 7722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:05.038126 7722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:05.039278 7722 hybrid_clock.cc:648] HybridClock initialized: now 1790263145039261 us; error 31 us; skew 500 ppm
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263145, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:19:05.041832 7722 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:19:05.042227 7722 webserver.cc:460] Webserver started at http://127.31.12.65:36373/ using document root <none> and password file <none>
I20260924 15:19:05.042408 7722 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:05.042451 7722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:05.042539 7722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:19:05.043550 7722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "6292a7b9e4f244b9a1430c886428b17e"
format_stamp: "Formatted at 2026-09-24 15:19:05 on dist-test-slave-zwbb"
server_key: "db0c7ced09e4fb94e2764406791c2d64"
server_key_iv: "08f11c664d2c81ba2a6dc9ea544c8bf6"
server_key_version: "encryptionkey@0"
I20260924 15:19:05.043860 7722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "6292a7b9e4f244b9a1430c886428b17e"
format_stamp: "Formatted at 2026-09-24 15:19:05 on dist-test-slave-zwbb"
server_key: "db0c7ced09e4fb94e2764406791c2d64"
server_key_iv: "08f11c664d2c81ba2a6dc9ea544c8bf6"
server_key_version: "encryptionkey@0"
I20260924 15:19:05.045125 7722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:19:05.045846 7737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:05.046025 7722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:05.046126 7722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "6292a7b9e4f244b9a1430c886428b17e"
format_stamp: "Formatted at 2026-09-24 15:19:05 on dist-test-slave-zwbb"
server_key: "db0c7ced09e4fb94e2764406791c2d64"
server_key_iv: "08f11c664d2c81ba2a6dc9ea544c8bf6"
server_key_version: "encryptionkey@0"
I20260924 15:19:05.046203 7722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:05.073791 7722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:05.075680 7722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:05.075843 7722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:05.076100 7722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:19:05.076417 7722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:19:05.076469 7722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:05.076509 7722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:19:05.076550 7722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:05.082630 7722 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:45147
I20260924 15:19:05.082685 7851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:45147 every 8 connection(s)
I20260924 15:19:05.083010 7722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263145, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:19:05.092044 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:05.083773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:47099 (local address 127.31.12.126:43993)
0924 15:19:05.083834 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:05.083836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:05.084054 (+ 218us) server_negotiation.cc:407] Connection header received
0924 15:19:05.084631 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:05.084632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:05.084645 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:05.084711 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:05.085689 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.086196 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.087181 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.087395 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.087472 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:05.089487 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:05.089488 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:05.089489 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:19:05.089496 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:05.090659 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:05.091069 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:05.091070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:05.091071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:05.091133 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:05.091369 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:05.091370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:05.091371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:05.091506 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:05.091578 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:05.091848 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 15:19:05.091950 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:19:05.092217 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:05.083747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.65:47099)
0924 15:19:05.083915 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:05.083975 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:05.084525 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:05.084789 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:05.084793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:05.085372 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:05.085589 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:05.085593 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.086295 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.086296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:05.087087 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:05.087089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.087493 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.087494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:05.087614 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:05.088048 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:05.088056 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:05.089360 (+ 1304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:19:05.090756 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:05.090764 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:05.090766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:05.090987 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:05.091198 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:05.091199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:05.091199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:05.091304 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:05.091580 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:19:05.091588 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:19:05.091720 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 15:19:05.091825 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 15:19:05.091982 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260924 15:19:05.092720 7852 heartbeater.cc:347] Connected to a master server at 127.31.12.126:43993
I20260924 15:19:05.092830 7852 heartbeater.cc:464] Registering TS with master...
I20260924 15:19:05.092887 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 7722
I20260924 15:19:05.093011 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:19:05.093034 7852 heartbeater.cc:511] Master 127.31.12.126:43993 requested a full tablet report, sending...
I20260924 15:19:05.093240 31793 external_mini_cluster.cc:1428] Setting key f12656c723ced1bec85c6e2c5336074e
I20260924 15:19:05.093632 7659 ts_manager.cc:194] Registered new tserver with Master: 6292a7b9e4f244b9a1430c886428b17e (127.31.12.65:45147)
I20260924 15:19:05.094364 7659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:47099
WARNING: no policy specified for kudu/127.31.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
I20260924 15:19:05.119704 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:43993
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:39943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:19:05.202898 7857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:05.203143 7857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:05.203182 7857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:05.204349 7857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:05.204416 7857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:05.204457 7857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:19:05.205922 7857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:39943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:05.206198 7857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:05.206422 7857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:05.211318 7863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:05.211272 7865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:05.211298 7857 server_base.cc:1034] running on GCE node
W20260924 15:19:05.211272 7862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:05.211788 7857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:05.212051 7857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:05.213186 7857 hybrid_clock.cc:648] HybridClock initialized: now 1790263145213172 us; error 29 us; skew 500 ppm
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263145, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:19:05.215773 7857 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:19:05.216192 7857 webserver.cc:460] Webserver started at http://127.31.12.66:37861/ using document root <none> and password file <none>
I20260924 15:19:05.216426 7857 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:05.216498 7857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:05.216637 7857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:19:05.217736 7857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "111820c51d734d1ca42763494edb357a"
format_stamp: "Formatted at 2026-09-24 15:19:05 on dist-test-slave-zwbb"
server_key: "f4fc3e2d2a6cbd3068b1871870bbbf98"
server_key_iv: "98c712fa1fbca4ea69e06a23c6216f1d"
server_key_version: "encryptionkey@0"
I20260924 15:19:05.218081 7857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "111820c51d734d1ca42763494edb357a"
format_stamp: "Formatted at 2026-09-24 15:19:05 on dist-test-slave-zwbb"
server_key: "f4fc3e2d2a6cbd3068b1871870bbbf98"
server_key_iv: "98c712fa1fbca4ea69e06a23c6216f1d"
server_key_version: "encryptionkey@0"
I20260924 15:19:05.219377 7857 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:19:05.220108 7872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:05.220269 7857 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:19:05.220366 7857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "111820c51d734d1ca42763494edb357a"
format_stamp: "Formatted at 2026-09-24 15:19:05 on dist-test-slave-zwbb"
server_key: "f4fc3e2d2a6cbd3068b1871870bbbf98"
server_key_iv: "98c712fa1fbca4ea69e06a23c6216f1d"
server_key_version: "encryptionkey@0"
I20260924 15:19:05.220436 7857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:05.232362 7857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:05.234520 7857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:05.234673 7857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:05.234895 7857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:19:05.235193 7857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:19:05.235247 7857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:05.235282 7857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:19:05.235308 7857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:05.240810 7857 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:33979
I20260924 15:19:05.240870 7986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:33979 every 8 connection(s)
I20260924 15:19:05.241189 7857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:19:05.244449 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 7857
I20260924 15:19:05.244556 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:19:05.244750 31793 external_mini_cluster.cc:1428] Setting key ded614070046971a429bad325a9195b2
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263145, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:19:05.251367 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:05.241963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:37313 (local address 127.31.12.126:43993)
0924 15:19:05.242026 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:05.242028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:05.242223 (+ 195us) server_negotiation.cc:407] Connection header received
0924 15:19:05.242744 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:05.242746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:05.242758 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:05.242813 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:05.243711 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.244264 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.245631 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.245832 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.245903 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:05.248357 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:05.248359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:05.248359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:19:05.248366 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:05.250035 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:05.250404 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:05.250406 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:05.250406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:05.250482 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:05.250706 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:05.250706 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:05.250707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:05.250867 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:05.250946 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:05.251175 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 15:19:05.251263 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 15:19:05.251392 7989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:05.241939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.66:37313)
0924 15:19:05.242114 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:05.242149 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:05.242646 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:05.242882 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:05.242886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:05.243424 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:05.243609 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:05.243613 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.244360 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.244361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:05.245554 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:05.245557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.245909 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.245910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:05.246048 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:05.246531 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:05.246540 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:05.248272 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:19:05.250115 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:05.250121 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:05.250123 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:05.250332 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:05.250555 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:05.250556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:05.250556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:05.250646 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:05.250933 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:19:05.250937 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:19:05.251048 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 15:19:05.251127 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 15:19:05.251236 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":88,"threads_started":1}
I20260924 15:19:05.251989 7987 heartbeater.cc:347] Connected to a master server at 127.31.12.126:43993
I20260924 15:19:05.252075 7987 heartbeater.cc:464] Registering TS with master...
I20260924 15:19:05.252251 7987 heartbeater.cc:511] Master 127.31.12.126:43993 requested a full tablet report, sending...
I20260924 15:19:05.252664 7659 ts_manager.cc:194] Registered new tserver with Master: 111820c51d734d1ca42763494edb357a (127.31.12.66:33979)
I20260924 15:19:05.253031 7659 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:37313
WARNING: no policy specified for kudu/127.31.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
I20260924 15:19:05.269872 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:43993
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:39943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:19:05.349215 7992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:05.349436 7992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:05.349501 7992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:05.350730 7992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:05.350790 7992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:05.350834 7992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:19:05.352237 7992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:39943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:05.352501 7992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:05.352710 7992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:05.357570 8000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:05.357585 7998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:05.357579 7997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:05.357750 7992 server_base.cc:1034] running on GCE node
I20260924 15:19:05.357980 7992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:05.358165 7992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:05.359319 7992 hybrid_clock.cc:648] HybridClock initialized: now 1790263145359304 us; error 22 us; skew 500 ppm
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263145, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:19:05.361693 7992 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:19:05.362119 7992 webserver.cc:460] Webserver started at http://127.31.12.67:37537/ using document root <none> and password file <none>
I20260924 15:19:05.362303 7992 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:05.362344 7992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:05.362429 7992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:19:05.363361 7992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "4d0441200f3c46b894d024f243d3240c"
format_stamp: "Formatted at 2026-09-24 15:19:05 on dist-test-slave-zwbb"
server_key: "d2ab358a0775b00b9623af836004a2dc"
server_key_iv: "1acbb0c030e9853a9a3869fbbacd25ea"
server_key_version: "encryptionkey@0"
I20260924 15:19:05.363664 7992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "4d0441200f3c46b894d024f243d3240c"
format_stamp: "Formatted at 2026-09-24 15:19:05 on dist-test-slave-zwbb"
server_key: "d2ab358a0775b00b9623af836004a2dc"
server_key_iv: "1acbb0c030e9853a9a3869fbbacd25ea"
server_key_version: "encryptionkey@0"
I20260924 15:19:05.364984 7992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:19:05.365680 8007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:05.365823 7992 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:19:05.365916 7992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "4d0441200f3c46b894d024f243d3240c"
format_stamp: "Formatted at 2026-09-24 15:19:05 on dist-test-slave-zwbb"
server_key: "d2ab358a0775b00b9623af836004a2dc"
server_key_iv: "1acbb0c030e9853a9a3869fbbacd25ea"
server_key_version: "encryptionkey@0"
I20260924 15:19:05.365985 7992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:05.406911 7992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:05.409044 7992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:05.409150 7992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:05.409325 7992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:19:05.409650 7992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:19:05.409685 7992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:05.409726 7992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:19:05.409751 7992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:05.415279 7992 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:42121
I20260924 15:19:05.415361 8123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:42121 every 8 connection(s)
I20260924 15:19:05.415656 7992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263145, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:19:05.424990 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 7992
I20260924 15:19:05.425088 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:19:05.425282 31793 external_mini_cluster.cc:1428] Setting key f8811fa02d5f9a21bc0985a94a2e88f6
I20260924 15:19:05.425289 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:05.416525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:41229 (local address 127.31.12.126:43993)
0924 15:19:05.416580 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:05.416582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:05.416813 (+ 231us) server_negotiation.cc:407] Connection header received
0924 15:19:05.417333 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:05.417335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:05.417348 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:05.417400 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:05.418340 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.418835 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.419812 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.420012 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.420098 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:05.422195 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:05.422196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:05.422196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:19:05.422203 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:05.423915 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:05.424369 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:05.424370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:05.424370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:05.424433 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:05.424639 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:05.424639 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:05.424639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:05.424740 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:05.424805 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:05.425144 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 15:19:05.425225 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 15:19:05.425625 8126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:05.416512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.67:41229)
0924 15:19:05.416683 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:05.416719 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:05.417225 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:05.417496 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:05.417500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:05.418057 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:05.418240 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:05.418244 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.418931 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.418932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:05.419708 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:05.419711 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.420107 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.420109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:05.420229 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:05.420673 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:05.420681 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:05.422100 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:19:05.424000 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:05.424008 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:05.424011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:05.424298 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:05.424491 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:05.424491 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:05.424491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:05.424585 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:05.424807 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:19:05.424811 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:19:05.424982 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 15:19:05.425097 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 15:19:05.425264 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260924 15:19:05.426048 8124 heartbeater.cc:347] Connected to a master server at 127.31.12.126:43993
I20260924 15:19:05.426163 8124 heartbeater.cc:464] Registering TS with master...
I20260924 15:19:05.426337 8124 heartbeater.cc:511] Master 127.31.12.126:43993 requested a full tablet report, sending...
I20260924 15:19:05.426666 7658 ts_manager.cc:194] Registered new tserver with Master: 4d0441200f3c46b894d024f243d3240c (127.31.12.67:42121)
I20260924 15:19:05.427053 7658 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:41229
I20260924 15:19:05.436872 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.65.keytab.
Entry for principal oryx/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.65.keytab.
WARNING: no policy specified for oryx/127.31.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.66.keytab.
Entry for principal oryx/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.66.keytab.
WARNING: no policy specified for oryx/127.31.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.31.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.67.keytab.
Entry for principal oryx/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.67.keytab.
I20260924 15:19:05.501012 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 7722
I20260924 15:19:05.505852 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 7857
I20260924 15:19:05.510245 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 7992
I20260924 15:19:05.514875 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 7627
I20260924 15:19:05.520213 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:43993
--webserver_interface=127.31.12.126
--webserver_port=35375
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:39943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:43993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:19:05.600320 8133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:05.600507 8133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:05.600528 8133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:05.601737 8133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:19:05.601776 8133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:05.601790 8133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:05.601837 8133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:19:05.601862 8133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:19:05.603376 8133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:39943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:43993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:43993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=35375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:05.603618 8133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:05.603835 8133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:05.608464 8138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:05.608453 8139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:05.608520 8133 server_base.cc:1034] running on GCE node
W20260924 15:19:05.608487 8141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:05.608901 8133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:05.609136 8133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:05.610267 8133 hybrid_clock.cc:648] HybridClock initialized: now 1790263145610255 us; error 28 us; skew 500 ppm
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263145, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:19:05.612649 8133 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:19:05.613073 8133 webserver.cc:460] Webserver started at http://127.31.12.126:35375/ using document root <none> and password file <none>
I20260924 15:19:05.613293 8133 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:05.613363 8133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:05.614708 8133 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:19:05.615362 8149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:05.615624 8133 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:19:05.615706 8133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "e8612f6c0a314205859f4ecb8997736b"
format_stamp: "Formatted at 2026-09-24 15:19:04 on dist-test-slave-zwbb"
server_key: "8f633462a6f79bb15e6bb660c419fc53"
server_key_iv: "eda57ada005de624eaebd14a66d0132b"
server_key_version: "encryptionkey@0"
I20260924 15:19:05.616024 8133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:05.622972 8133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:05.625072 8133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:05.625175 8133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:05.628880 8133 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:43993
I20260924 15:19:05.628957 8201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:43993 every 8 connection(s)
I20260924 15:19:05.629294 8133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:19:05.630707 8202 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:19:05.631247 8202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b: Bootstrap starting.
I20260924 15:19:05.635077 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 8133
I20260924 15:19:05.635424 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:45147
--local_ip_for_outbound_sockets=127.31.12.65
--tserver_master_addrs=127.31.12.126:43993
--webserver_port=36373
--webserver_interface=127.31.12.65
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:39943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.12.65
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:19:05.639951 8202 log.cc:824] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b: Log is configured to *not* fsync() on all Append() calls
I20260924 15:19:05.641587 8202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:19:05.641861 8202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b: Bootstrap complete.
I20260924 15:19:05.644062 8202 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8612f6c0a314205859f4ecb8997736b" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43993 } }
I20260924 15:19:05.644300 8202 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8612f6c0a314205859f4ecb8997736b, State: Initialized, Role: FOLLOWER
I20260924 15:19:05.644506 8202 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8612f6c0a314205859f4ecb8997736b" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43993 } }
I20260924 15:19:05.644598 8202 raft_consensus.cc:388] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:19:05.644654 8202 raft_consensus.cc:482] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:19:05.644702 8202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [term 1 FOLLOWER]: Advancing to term 2
I20260924 15:19:05.645745 8202 raft_consensus.cc:504] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8612f6c0a314205859f4ecb8997736b" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43993 } }
I20260924 15:19:05.645857 8202 leader_election.cc:302] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8612f6c0a314205859f4ecb8997736b; no voters:
I20260924 15:19:05.646040 8202 leader_election.cc:288] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 15:19:05.646075 8206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [term 2 FOLLOWER]: Leader election won for term 2
I20260924 15:19:05.646198 8206 raft_consensus.cc:686] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [term 2 LEADER]: Becoming Leader. State: Replica: e8612f6c0a314205859f4ecb8997736b, State: Running, Role: LEADER
I20260924 15:19:05.646313 8202 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:19:05.646379 8206 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8612f6c0a314205859f4ecb8997736b" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43993 } }
I20260924 15:19:05.646677 8206 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8612f6c0a314205859f4ecb8997736b. Latest consensus state: current_term: 2 leader_uuid: "e8612f6c0a314205859f4ecb8997736b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8612f6c0a314205859f4ecb8997736b" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43993 } } }
I20260924 15:19:05.646764 8206 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [sys.catalog]: This master's current role is: LEADER
I20260924 15:19:05.646907 8207 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e8612f6c0a314205859f4ecb8997736b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8612f6c0a314205859f4ecb8997736b" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43993 } } }
I20260924 15:19:05.646978 8207 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b [sys.catalog]: This master's current role is: LEADER
I20260924 15:19:05.648125 8222 catalog_manager.cc:1259] Loaded cluster ID: 1c1b5f75607747da889f112238ff8827
I20260924 15:19:05.648185 8222 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b: loading cluster ID for follower catalog manager: success
I20260924 15:19:05.648955 8222 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b: acquiring CA information for follower catalog manager: success
I20260924 15:19:05.649313 8222 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:19:05.649386 8213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:19:05.649533 8213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:19:05.649613 8213 catalog_manager.cc:1259] Loaded cluster ID: 1c1b5f75607747da889f112238ff8827
I20260924 15:19:05.649645 8213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:19:05.649793 8213 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:19:05.649880 8213 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e8612f6c0a314205859f4ecb8997736b: Loaded TSK: 0
I20260924 15:19:05.650027 8213 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 15:19:05.737942 8204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:05.738145 8204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:05.738169 8204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:05.739351 8204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:05.739393 8204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:05.739420 8204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:19:05.740899 8204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:39943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.65.keytab
--principal=oryx/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:45147
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=36373
--tserver_master_addrs=127.31.12.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:05.741123 8204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:05.741357 8204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:05.747627 8228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:05.747603 8229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:05.747594 8231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:05.748024 8204 server_base.cc:1034] running on GCE node
I20260924 15:19:05.748255 8204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:05.748550 8204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:05.749732 8204 hybrid_clock.cc:648] HybridClock initialized: now 1790263145749717 us; error 27 us; skew 500 ppm
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263145, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:19:05.756398 8204 init.cc:377] Logged in from keytab as oryx/127.31.12.65@KRBTEST.COM (short username oryx)
I20260924 15:19:05.756863 8204 webserver.cc:460] Webserver started at http://127.31.12.65:36373/ using document root <none> and password file <none>
I20260924 15:19:05.757091 8204 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:05.757167 8204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:05.758594 8204 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:19:05.759265 8238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:05.759436 8204 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:19:05.759538 8204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "6292a7b9e4f244b9a1430c886428b17e"
format_stamp: "Formatted at 2026-09-24 15:19:05 on dist-test-slave-zwbb"
server_key: "db0c7ced09e4fb94e2764406791c2d64"
server_key_iv: "08f11c664d2c81ba2a6dc9ea544c8bf6"
server_key_version: "encryptionkey@0"
I20260924 15:19:05.759888 8204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:05.773327 8204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:05.775468 8204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:05.775619 8204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:05.775832 8204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:19:05.776129 8204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:19:05.776178 8204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:05.776219 8204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:19:05.776253 8204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:05.782544 8204 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:45147
I20260924 15:19:05.782588 8353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:45147 every 8 connection(s)
I20260924 15:19:05.782904 8204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:05.791105 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 8204
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:05.791422 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:33979
--local_ip_for_outbound_sockets=127.31.12.66
--tserver_master_addrs=127.31.12.126:43993
--webserver_port=37861
--webserver_interface=127.31.12.66
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:39943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.12.66
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:19:05.792286 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:05.783612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.65:47983)
0924 15:19:05.783776 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:05.783810 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:05.784351 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:05.784903 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:05.784907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:05.785682 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:05.785955 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:05.785962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.787046 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.787047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:05.788235 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:05.788237 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.788784 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.788785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:05.788924 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:05.789411 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:05.789420 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:05.792053 (+ 2633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
W20260924 15:19:05.792531 8357 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:05.783896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:47983 (local address 127.31.12.126:43993)
0924 15:19:05.784104 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:05.784113 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:05.784153 (+ 40us) server_negotiation.cc:407] Connection header received
0924 15:19:05.784653 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:05.784658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:05.784698 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:05.784793 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:05.786080 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.786951 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.788317 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.788705 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.788782 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:05.792219 (+ 3437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.65:47983: BlockingRecv error: recv got EOF from 127.31.12.65:47983 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":94,"threads_started":1}
W20260924 15:19:05.792805 8354 heartbeater.cc:644] Failed to heartbeat to 127.31.12.126:43993 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.12.126:43993: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:05.797082 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:05.793067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.65:54873)
0924 15:19:05.793109 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:05.793119 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:05.793157 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:05.793290 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:05.793291 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:05.793412 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:05.793492 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:05.793494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.794119 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.794120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:05.794629 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:05.794630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.794965 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.794966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:05.795043 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:05.795417 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:05.795422 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:05.797006 (+ 1584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 15:19:05.797152 8357 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:05.793100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:54873 (local address 127.31.12.126:43993)
0924 15:19:05.793137 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:05.793138 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:05.793146 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:19:05.793206 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:05.793206 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:05.793218 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:05.793288 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:05.793577 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.794024 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.794703 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.794889 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.794955 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:05.797085 (+ 2130us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.65:54873: BlockingRecv error: recv got EOF from 127.31.12.65:54873 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:05.801465 8356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:05.797577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.65:33111)
0924 15:19:05.797658 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:05.797669 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:05.797732 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:05.797853 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:05.797854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:05.797971 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:05.797996 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:05.797997 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.798582 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.798583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:05.799060 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:05.799061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.799400 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.799401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:05.799454 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:05.799837 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:05.799842 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:05.801371 (+ 1529us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260924 15:19:05.801523 8357 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:05.797622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:33111 (local address 127.31.12.126:43993)
0924 15:19:05.797666 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:05.797668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:05.797718 (+ 50us) server_negotiation.cc:407] Connection header received
0924 15:19:05.797788 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:05.797789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:05.797799 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:05.797841 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:05.798065 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.798504 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.799145 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.799322 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.799397 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:05.801436 (+ 2039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.65:33111: BlockingRecv error: recv got EOF from 127.31.12.65:33111 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 15:19:05.801829 8354 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 15:19:05.886480 8358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:05.886711 8358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:05.886737 8358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:05.887981 8358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:05.888023 8358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:05.888052 8358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:19:05.889585 8358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:39943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.66.keytab
--principal=oryx/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:33979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=37861
--tserver_master_addrs=127.31.12.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:05.889776 8358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:05.889971 8358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:05.895147 8364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:05.895162 8363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:05.895201 8358 server_base.cc:1034] running on GCE node
W20260924 15:19:05.895150 8366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:05.895730 8358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:05.895937 8358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:05.897094 8358 hybrid_clock.cc:648] HybridClock initialized: now 1790263145897079 us; error 29 us; skew 500 ppm
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263145, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:19:05.899546 8358 init.cc:377] Logged in from keytab as oryx/127.31.12.66@KRBTEST.COM (short username oryx)
I20260924 15:19:05.899953 8358 webserver.cc:460] Webserver started at http://127.31.12.66:37861/ using document root <none> and password file <none>
I20260924 15:19:05.900175 8358 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:05.900252 8358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:05.901652 8358 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:19:05.902220 8373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:05.902390 8358 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:05.902483 8358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "111820c51d734d1ca42763494edb357a"
format_stamp: "Formatted at 2026-09-24 15:19:05 on dist-test-slave-zwbb"
server_key: "f4fc3e2d2a6cbd3068b1871870bbbf98"
server_key_iv: "98c712fa1fbca4ea69e06a23c6216f1d"
server_key_version: "encryptionkey@0"
I20260924 15:19:05.902863 8358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:05.911429 8358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:05.913748 8358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:05.913910 8358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:05.914156 8358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:19:05.914458 8358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:19:05.914510 8358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:05.914551 8358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:19:05.914582 8358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:05.920539 8358 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:33979
I20260924 15:19:05.920584 8487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:33979 every 8 connection(s)
I20260924 15:19:05.920940 8358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:05.927206 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 8358
I20260924 15:19:05.927479 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:42121
--local_ip_for_outbound_sockets=127.31.12.67
--tserver_master_addrs=127.31.12.126:43993
--webserver_port=37537
--webserver_interface=127.31.12.67
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:39943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.31.12.67
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:05.928367 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:05.921702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.66:43761)
0924 15:19:05.921901 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:05.921939 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:05.922454 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:05.922713 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:05.922717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:05.923468 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:05.923658 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:05.923662 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.924352 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.924353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:05.925179 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:05.925181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.925621 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.925622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:05.925742 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:05.926173 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:05.926181 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:05.928225 (+ 2044us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":77,"threads_started":1}
W20260924 15:19:05.928426 8357 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:05.921708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:43761 (local address 127.31.12.126:43993)
0924 15:19:05.921820 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:05.921822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:05.922027 (+ 205us) server_negotiation.cc:407] Connection header received
0924 15:19:05.922569 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:05.922570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:05.922588 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:05.922642 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:05.923771 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.924261 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.925269 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.925532 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.925611 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:05.928339 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.66:43761: BlockingRecv error: recv got EOF from 127.31.12.66:43761 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 15:19:05.928668 8488 heartbeater.cc:644] Failed to heartbeat to 127.31.12.126:43993 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.12.126:43993: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:05.933024 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:05.928897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.66:44863)
0924 15:19:05.928942 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:05.928953 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:05.928998 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:05.929152 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:05.929153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:05.929272 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:05.929313 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:05.929314 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.929957 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.929959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:05.930451 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:05.930453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.930798 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.930799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:05.930868 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:05.931308 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:05.931312 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:05.932981 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 15:19:05.933171 8357 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:05.928923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:44863 (local address 127.31.12.126:43993)
0924 15:19:05.928961 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:05.928963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:05.928986 (+ 23us) server_negotiation.cc:407] Connection header received
0924 15:19:05.929060 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:05.929060 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:05.929069 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:05.929145 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:05.929392 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.929880 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.930527 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.930720 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.930798 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:05.933094 (+ 2296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.66:44863: BlockingRecv error: recv got EOF from 127.31.12.66:44863 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:05 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:05.937978 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:05.933379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.66:53799)
0924 15:19:05.933434 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:05.933465 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:05.933516 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:05.933656 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:05.933657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:05.933812 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:05.933850 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:05.933852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.934410 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.934411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:05.935301 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:05.935304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.935616 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.935617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:05.935683 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:05.936047 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:05.936053 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:05.937886 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 15:19:05.937994 8357 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:05.933429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:53799 (local address 127.31.12.126:43993)
0924 15:19:05.933487 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:05.933489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:05.933503 (+ 14us) server_negotiation.cc:407] Connection header received
0924 15:19:05.933586 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:05.933587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:05.933594 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:05.933636 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:05.933915 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.934339 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.935379 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:05.935550 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:05.935598 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:05.937957 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.66:53799: BlockingRecv error: recv got EOF from 127.31.12.66:53799 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 15:19:05.938102 8488 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 15:19:06.009791 8491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:06.009975 8491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:06.010003 8491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:06.011092 8491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:06.011127 8491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:06.011152 8491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:19:06.012552 8491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:39943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/krb5kdc/oryx_127.31.12.67.keytab
--principal=oryx/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:42121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=37537
--tserver_master_addrs=127.31.12.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:06.012732 8491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:06.012895 8491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:06.017918 8497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:06.017969 8496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:06.017974 8491 server_base.cc:1034] running on GCE node
W20260924 15:19:06.017915 8499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:06.018339 8491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:06.018621 8491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:06.019774 8491 hybrid_clock.cc:648] HybridClock initialized: now 1790263146019762 us; error 30 us; skew 500 ppm
Sep 24 15:19:06 dist-test-slave-zwbb krb5kdc[7614](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263146, etypes {rep=17 tkt=17 ses=17}, oryx/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:19:06.022306 8491 init.cc:377] Logged in from keytab as oryx/127.31.12.67@KRBTEST.COM (short username oryx)
I20260924 15:19:06.022681 8491 webserver.cc:460] Webserver started at http://127.31.12.67:37537/ using document root <none> and password file <none>
I20260924 15:19:06.022876 8491 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:06.022925 8491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:06.024183 8491 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:19:06.024693 8506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:06.024855 8491 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:19:06.024950 8491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "4d0441200f3c46b894d024f243d3240c"
format_stamp: "Formatted at 2026-09-24 15:19:05 on dist-test-slave-zwbb"
server_key: "d2ab358a0775b00b9623af836004a2dc"
server_key_iv: "1acbb0c030e9853a9a3869fbbacd25ea"
server_key_version: "encryptionkey@0"
I20260924 15:19:06.025290 8491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:06.037674 8491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:06.039888 8491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:06.040047 8491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:06.040268 8491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:19:06.040565 8491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:19:06.040614 8491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:06.040654 8491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:19:06.040683 8491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:06.046458 8491 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:42121
I20260924 15:19:06.046507 8621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:42121 every 8 connection(s)
I20260924 15:19:06.046882 8491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:19:06 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:06.052696 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 8491
Sep 24 15:19:06 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
W20260924 15:19:06.054117 8357 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:06.047577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:46539 (local address 127.31.12.126:43993)
0924 15:19:06.047623 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:06.047625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:06.047858 (+ 233us) server_negotiation.cc:407] Connection header received
0924 15:19:06.048373 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:06.048374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:06.048386 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:06.048441 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:06.049360 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.049921 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.050891 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.051100 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.051178 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:06.054053 (+ 2875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.67:46539: BlockingRecv error: recv got EOF from 127.31.12.67:46539 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:19:06.054199 8624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:06.047575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.67:46539)
0924 15:19:06.047744 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:06.047778 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:06.048275 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:06.048519 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:06.048522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:06.049068 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:06.049255 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:06.049260 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.050013 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.050015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:06.050795 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:06.050796 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.051189 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.051190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:06.051310 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:06.051727 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:06.051734 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:06.053997 (+ 2263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
W20260924 15:19:06.054416 8622 heartbeater.cc:644] Failed to heartbeat to 127.31.12.126:43993 (0 consecutive failures): Not authorized: Failed to ping master at 127.31.12.126:43993: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Sep 24 15:19:06 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:06 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:06.059386 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:06.054641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.67:44085)
0924 15:19:06.054763 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:06.054773 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:06.054833 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:06.054969 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:06.054969 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:06.055102 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:06.055142 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:06.055143 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.055961 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.055962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:06.056745 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:06.056747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.057209 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.057210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:06.057252 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:06.057664 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:06.057669 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:06.059299 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
W20260924 15:19:06.059428 8357 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:06.054680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:44085 (local address 127.31.12.126:43993)
0924 15:19:06.054717 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:06.054720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:06.054823 (+ 103us) server_negotiation.cc:407] Connection header received
0924 15:19:06.054898 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:06.054899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:06.054909 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:06.054959 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:06.055204 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.055898 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.056822 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.057116 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.057196 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:06.059365 (+ 2169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.67:44085: BlockingRecv error: recv got EOF from 127.31.12.67:44085 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:19:06.060567 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:06.053220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33904 (local address 127.31.12.126:43993)
0924 15:19:06.053322 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:06.053324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:06.053337 (+ 13us) server_negotiation.cc:407] Connection header received
0924 15:19:06.053399 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:06.053400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:06.053411 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:06.053499 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:06.054066 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.054807 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.055882 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.056173 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.056239 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:06.057587 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:06.057591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:06.057592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:19:06.057602 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:06.059207 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:06.059583 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:06.059587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:06.059587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:06.059702 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:06.059881 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:06.059882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:06.059882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:06.060111 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:06.060253 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:06.060362 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 15:19:06.060474 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":66,"threads_started":1}
Sep 24 15:19:06 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:06 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:06.063589 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:06.059609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.67:56317)
0924 15:19:06.059647 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:06.059656 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:06.059690 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:06.059811 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:06.059812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:06.059930 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:06.059952 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:06.059953 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.060565 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.060566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:06.061242 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:06.061244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.061691 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.061691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:06.061731 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:06.062046 (+ 315us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:06.062050 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:06.063536 (+ 1486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 15:19:06.063623 8357 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:06.059649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:56317 (local address 127.31.12.126:43993)
0924 15:19:06.059684 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:06.059685 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:06.059693 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:19:06.059745 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:06.059746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:06.059755 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:06.059808 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:06.060024 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.060428 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.061327 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.061633 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.061684 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:06.063588 (+ 1904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.67:56317: BlockingRecv error: recv got EOF from 127.31.12.67:56317 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 15:19:06.063716 8622 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 24 15:19:06 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:06 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:06.807093 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:06.802268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.65:57287)
0924 15:19:06.802430 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:06.802440 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:06.802490 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:06.802627 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:06.802628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:06.802859 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:06.802939 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:06.802942 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.803747 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.803748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:06.804287 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:06.804288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.804600 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.804601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:06.804641 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:06.805055 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:06.805059 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:06.807021 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":94,"threads_started":1}
W20260924 15:19:06.807225 8629 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:06.802346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:57287 (local address 127.31.12.126:43993)
0924 15:19:06.802472 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:06.802473 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:06.802482 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:19:06.802538 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:06.802538 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:06.802548 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:06.802611 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:06.803023 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.803679 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.804348 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.804540 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.804588 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:06.807130 (+ 2542us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.65:57287: BlockingRecv error: recv got EOF from 127.31.12.65:57287 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
Sep 24 15:19:06 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:06 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:06.942927 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:06.938584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.66:60677)
0924 15:19:06.938717 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:06.938727 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:06.938782 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:06.938898 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:06.938899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:06.939061 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:06.939110 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:06.939111 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.939779 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.939779 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:06.940302 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:06.940303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.940648 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.940649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:06.940700 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:06.941153 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:06.941158 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:06.942865 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
W20260924 15:19:06.943086 8629 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:06.938651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:60677 (local address 127.31.12.126:43993)
0924 15:19:06.938735 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:06.938737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:06.938780 (+ 43us) server_negotiation.cc:407] Connection header received
0924 15:19:06.938824 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:06.938825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:06.938836 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:06.938888 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:06.939166 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.939712 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.940369 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:06.940588 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:06.940642 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:06.943004 (+ 2362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.66:60677: BlockingRecv error: recv got EOF from 127.31.12.66:60677 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 15:19:07 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:07 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:07.068550 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:07.064045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.67:35963)
0924 15:19:07.064202 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:07.064214 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:07.064280 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:07.064443 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:07.064444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:07.064628 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:07.064672 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:07.064674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:07.065343 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:07.065344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:07.065947 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:07.065949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:07.066316 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:07.066317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:07.066382 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:07.066799 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:07.066803 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:07.068443 (+ 1640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
W20260924 15:19:07.068590 8629 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:07.064119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:35963 (local address 127.31.12.126:43993)
0924 15:19:07.064191 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:07.064194 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:07.064261 (+ 67us) server_negotiation.cc:407] Connection header received
0924 15:19:07.064334 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:07.064334 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:07.064349 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:07.064436 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:07.064749 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:07.065263 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:07.066035 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:07.066224 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:07.066316 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:07.068526 (+ 2210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.67:35963: BlockingRecv error: recv got EOF from 127.31.12.67:35963 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 15:19:07 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:07 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:07.811920 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:07.807536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.65:42263)
0924 15:19:07.807660 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:07.807670 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:07.807715 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:07.807854 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:07.807854 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:07.808007 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:07.808057 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:07.808058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:07.808672 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:07.808672 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:07.809172 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:07.809173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:07.809570 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:07.809570 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:07.809610 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:07.810049 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:07.810053 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:07.811845 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
W20260924 15:19:07.812041 8633 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:07.807601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:42263 (local address 127.31.12.126:43993)
0924 15:19:07.807718 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:07.807720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:07.807730 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:19:07.807761 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:07.807762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:07.807772 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:07.807835 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:07.808114 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:07.808611 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:07.809234 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:07.809414 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:07.809493 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:07.811930 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.65:42263: BlockingRecv error: recv got EOF from 127.31.12.65:42263 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
Sep 24 15:19:07 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:07 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:07.948180 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:07.943525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.66:50019)
0924 15:19:07.943657 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:07.943667 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:07.943723 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:07.943860 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:07.943861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:07.944028 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:07.944076 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:07.944078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:07.944690 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:07.944691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:07.945263 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:07.945266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:07.945719 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:07.945721 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:07.945785 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:07.946305 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:07.946310 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:07.948087 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":67,"threads_started":1}
W20260924 15:19:07.948221 8633 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:07.943589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:50019 (local address 127.31.12.126:43993)
0924 15:19:07.943659 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:07.943667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:07.943708 (+ 41us) server_negotiation.cc:407] Connection header received
0924 15:19:07.943787 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:07.943787 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:07.943799 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:07.943841 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:07.944135 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:07.944624 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:07.945346 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:07.945622 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:07.945716 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:07.948160 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.66:50019: BlockingRecv error: recv got EOF from 127.31.12.66:50019 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 15:19:08 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:08 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:08.073189 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:08.069007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.67:43375)
0924 15:19:08.069164 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:08.069180 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:08.069234 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:08.069356 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:08.069357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:08.069540 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:08.069588 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:08.069589 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:08.070194 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:08.070195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:08.070721 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:08.070723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:08.071044 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:08.071045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:08.071084 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:08.071499 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:08.071503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:08.073133 (+ 1630us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":59,"threads_started":1}
W20260924 15:19:08.073276 8633 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:08.069186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:43375 (local address 127.31.12.126:43993)
0924 15:19:08.069243 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:08.069249 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:08.069270 (+ 21us) server_negotiation.cc:407] Connection header received
0924 15:19:08.069292 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:08.069292 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:08.069303 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:08.069342 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:08.069647 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:08.070130 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:08.070798 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:08.070975 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:08.071034 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:08.073212 (+ 2178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.67:43375: BlockingRecv error: recv got EOF from 127.31.12.67:43375 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 15:19:08 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:08 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:08.817199 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:08.812386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.65:38759)
0924 15:19:08.812523 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:08.812533 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:08.812590 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:08.812762 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:08.812763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:08.812925 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:08.812979 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:08.812981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:08.813636 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:08.813637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:08.814162 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:08.814164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:08.814692 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:08.814693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:08.814764 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:08.815251 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:08.815256 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:08.817088 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
W20260924 15:19:08.817252 8637 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:08.812463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:38759 (local address 127.31.12.126:43993)
0924 15:19:08.812607 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:08.812609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:08.812623 (+ 14us) server_negotiation.cc:407] Connection header received
0924 15:19:08.812645 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:08.812646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:08.812655 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:08.812723 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:08.813045 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:08.813553 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:08.814242 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:08.814593 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:08.814692 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:08.817169 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.65:38759: BlockingRecv error: recv got EOF from 127.31.12.65:38759 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
Sep 24 15:19:08 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:08 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:08.953082 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:08.948626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.66:53549)
0924 15:19:08.948779 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:08.948790 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:08.948852 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:08.948981 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:08.948982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:08.949142 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:08.949191 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:08.949192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:08.949905 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:08.949907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:08.950440 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:08.950441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:08.950781 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:08.950782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:08.950818 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:08.951391 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:08.951395 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:08.953016 (+ 1621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
W20260924 15:19:08.953228 8637 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:08.948671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:53549 (local address 127.31.12.126:43993)
0924 15:19:08.948732 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:08.948735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:08.948850 (+ 115us) server_negotiation.cc:407] Connection header received
0924 15:19:08.948903 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:08.948904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:08.948916 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:08.948964 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:08.949279 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:08.949815 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:08.950535 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:08.950729 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:08.950774 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:08.953109 (+ 2335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.66:53549: BlockingRecv error: recv got EOF from 127.31.12.66:53549 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 15:19:09 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:09 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:09.078305 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:09.073689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.67:51653)
0924 15:19:09.073836 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:09.073846 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:09.073904 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:09.074040 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:09.074040 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:09.074201 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:09.074248 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:09.074250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:09.074851 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:09.074851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:09.075410 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:09.075412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:09.075901 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:09.075902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:09.075948 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:09.076369 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:09.076374 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:09.078208 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
W20260924 15:19:09.078349 8637 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:09.073767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:51653 (local address 127.31.12.126:43993)
0924 15:19:09.073848 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:09.073850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:09.073894 (+ 44us) server_negotiation.cc:407] Connection header received
0924 15:19:09.073948 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:09.073948 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:09.073960 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:09.074035 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:09.074316 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:09.074787 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:09.075618 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:09.075807 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:09.075886 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:09.078285 (+ 2399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.67:51653: BlockingRecv error: recv got EOF from 127.31.12.67:51653 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 15:19:09 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:09 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:09.822296 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:09.817758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.65:32939)
0924 15:19:09.817940 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:09.817951 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:09.818017 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:09.818173 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:09.818174 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:09.818324 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:09.818375 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:09.818376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:09.819055 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:09.819056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:09.819568 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:09.819570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:09.819910 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:09.819911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:09.819962 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:09.820447 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:09.820452 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:09.822242 (+ 1790us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":85,"threads_started":1}
W20260924 15:19:09.822436 8642 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:09.817822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:32939 (local address 127.31.12.126:43993)
0924 15:19:09.817948 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:09.817949 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:09.818012 (+ 63us) server_negotiation.cc:407] Connection header received
0924 15:19:09.818061 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:09.818061 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:09.818071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:09.818160 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:09.818461 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:09.818967 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:09.819637 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:09.819842 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:09.819892 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:09.822341 (+ 2449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.65:32939: BlockingRecv error: recv got EOF from 127.31.12.65:32939 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":45,"threads_started":1}
Sep 24 15:19:09 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:09 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:09.957674 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:09.953486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.66:42585)
0924 15:19:09.953644 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:09.953656 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:09.953720 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:09.953927 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:09.953929 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:09.954079 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:09.954128 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:09.954130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:09.954732 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:09.954732 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:09.955247 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:09.955249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:09.955567 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:09.955568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:09.955606 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:09.956014 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:09.956019 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:09.957617 (+ 1598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
W20260924 15:19:09.957790 8642 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:09.953736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:42585 (local address 127.31.12.126:43993)
0924 15:19:09.953797 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:09.953799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:09.953806 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:19:09.953828 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:09.953829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:09.953839 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:09.953888 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:09.954185 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:09.954664 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:09.955317 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:09.955506 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:09.955561 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:09.957699 (+ 2138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.66:42585: BlockingRecv error: recv got EOF from 127.31.12.66:42585 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 15:19:10 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:10 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:10.084146 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:10.078817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.67:40615)
0924 15:19:10.078950 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:10.078962 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:10.079033 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:10.079201 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:10.079203 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:10.079475 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:10.079545 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:10.079547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:10.080371 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:10.080372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:10.081135 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:10.081138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:10.081616 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:10.081617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:10.081688 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:10.082138 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:10.082143 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:10.084049 (+ 1906us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":89,"threads_started":1}
W20260924 15:19:10.084193 8642 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:10.078913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:40615 (local address 127.31.12.126:43993)
0924 15:19:10.078959 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:10.078962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:10.079018 (+ 56us) server_negotiation.cc:407] Connection header received
0924 15:19:10.079080 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:10.079080 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:10.079094 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:10.079151 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:10.079631 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:10.080294 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:10.081214 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:10.081391 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:10.081491 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:10.084128 (+ 2637us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.67:40615: BlockingRecv error: recv got EOF from 127.31.12.67:40615 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 15:19:10 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:10 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:10.831770 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:10.822732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.65:35477)
0924 15:19:10.822873 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:10.822883 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:10.822938 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:10.823086 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:10.823087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:10.823237 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:10.823288 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:10.823289 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:10.823875 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:10.823875 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:10.824397 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:10.824405 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:10.824722 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:10.824723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:10.824764 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:10.829908 (+ 5144us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:10.829914 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:10.831711 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
W20260924 15:19:10.831905 8646 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:10.822806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:35477 (local address 127.31.12.126:43993)
0924 15:19:10.822924 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:10.822926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:10.822934 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:19:10.823005 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:10.823005 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:10.823016 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:10.823071 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:10.823343 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:10.823813 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:10.824474 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:10.824657 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:10.824707 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:10.831816 (+ 7109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.65:35477: BlockingRecv error: recv got EOF from 127.31.12.65:35477 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":65,"threads_started":1}
Sep 24 15:19:10 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:10 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:10.962446 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:10.958088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.66:48177)
0924 15:19:10.958257 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:10.958268 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:10.958323 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:10.958446 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:10.958447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:10.958605 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:10.958653 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:10.958655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:10.959274 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:10.959275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:10.959773 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:10.959774 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:10.960068 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:10.960069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:10.960106 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:10.960507 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:10.960511 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:10.962354 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":54,"threads_started":1}
W20260924 15:19:10.962512 8646 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:10.958196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:48177 (local address 127.31.12.126:43993)
0924 15:19:10.958259 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:10.958261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:10.958322 (+ 61us) server_negotiation.cc:407] Connection header received
0924 15:19:10.958372 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:10.958372 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:10.958383 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:10.958428 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:10.958712 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:10.959198 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:10.959837 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:10.960011 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:10.960056 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:10.962450 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.66:48177: BlockingRecv error: recv got EOF from 127.31.12.66:48177 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 15:19:11 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:11 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:11.089353 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:11.084770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.67:36573)
0924 15:19:11.084948 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:11.084962 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:11.085025 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:11.085203 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:11.085204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:11.085400 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:11.085492 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:11.085495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:11.086154 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:11.086155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:11.086689 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:11.086691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:11.087061 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:11.087062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:11.087127 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:11.087549 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:11.087553 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:11.089258 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":119,"threads_started":1}
W20260924 15:19:11.089406 8646 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:11.084933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:36573 (local address 127.31.12.126:43993)
0924 15:19:11.085007 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:11.085008 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:11.085017 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:19:11.085120 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:11.085121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:11.085133 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:11.085195 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:11.085578 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:11.086071 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:11.086771 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:11.086977 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:11.087050 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:11.089342 (+ 2292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.67:36573: BlockingRecv error: recv got EOF from 127.31.12.67:36573 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 24 15:19:11 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:11 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:11.836834 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:11.832255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.65:40177)
0924 15:19:11.832429 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:11.832442 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:11.832511 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:11.832648 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:11.832649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:11.832808 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:11.832860 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:11.832862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:11.833556 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:11.833557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:11.834136 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:11.834139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:11.834460 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:11.834462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:11.834510 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:11.834941 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:11.834945 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:11.836771 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":93,"threads_started":1}
W20260924 15:19:11.836937 8650 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:11.832345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:40177 (local address 127.31.12.126:43993)
0924 15:19:11.832465 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:11.832467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:11.832500 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:19:11.832562 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:11.832562 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:11.832572 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:11.832630 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:11.832930 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:11.833417 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:11.834206 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:11.834393 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:11.834441 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:11.836868 (+ 2427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.65:40177: BlockingRecv error: recv got EOF from 127.31.12.65:40177 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
Sep 24 15:19:11 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:11 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:11.967389 8651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:11.962933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.66:33071)
0924 15:19:11.963056 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:11.963074 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:11.963127 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:11.963253 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:11.963254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:11.963423 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:11.963472 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:11.963474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:11.964113 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:11.964114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:11.964718 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:11.964721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:11.965050 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:11.965051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:11.965091 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:11.965563 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:11.965567 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:11.967299 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
W20260924 15:19:11.967422 8650 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:11.963025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:33071 (local address 127.31.12.126:43993)
0924 15:19:11.963075 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:11.963076 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:11.963113 (+ 37us) server_negotiation.cc:407] Connection header received
0924 15:19:11.963176 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:11.963176 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:11.963186 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:11.963237 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:11.963531 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:11.964042 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:11.964789 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:11.964977 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:11.965035 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:11.967364 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.66:33071: BlockingRecv error: recv got EOF from 127.31.12.66:33071 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 15:19:12 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:12 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:12.094249 8652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:12.089853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.67:36225)
0924 15:19:12.090044 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:12.090055 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:12.090121 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:12.090254 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:12.090255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:12.090424 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:12.090473 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:12.090474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:12.091102 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:12.091102 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:12.091618 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:12.091620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:12.091935 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:12.091936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:12.091974 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:12.092460 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:12.092469 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:12.094189 (+ 1720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":134,"threads_started":1}
W20260924 15:19:12.094372 8650 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:12.089958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:36225 (local address 127.31.12.126:43993)
0924 15:19:12.090038 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:12.090040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:12.090123 (+ 83us) server_negotiation.cc:407] Connection header received
0924 15:19:12.090178 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:12.090179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:12.090191 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:12.090234 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:12.090539 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:12.091039 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:12.091688 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:12.091869 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:12.091936 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:12.094279 (+ 2343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.67:36225: BlockingRecv error: recv got EOF from 127.31.12.67:36225 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 15:19:12 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:12 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:12.842219 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:12.837426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.65:58397)
0924 15:19:12.837604 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:12.837614 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:12.837680 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:12.837814 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:12.837814 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:12.837990 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:12.838040 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:12.838042 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:12.838750 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:12.838751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:12.839292 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:12.839293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:12.839767 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:12.839768 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:12.839854 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:12.840354 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:12.840359 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:12.842161 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":113,"threads_started":1}
W20260924 15:19:12.842396 8654 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:12.837520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:58397 (local address 127.31.12.126:43993)
0924 15:19:12.837676 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:12.837677 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:12.837687 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:19:12.837720 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:12.837720 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:12.837731 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:12.837798 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:12.838112 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:12.838677 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:12.839424 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:12.839666 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:12.839757 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:12.842295 (+ 2538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.65:58397: BlockingRecv error: recv got EOF from 127.31.12.65:58397 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
Sep 24 15:19:12 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:12 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:12.972424 8655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:12.967863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.66:55653)
0924 15:19:12.967999 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:12.968010 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:12.968076 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:12.968209 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:12.968210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:12.968363 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:12.968422 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:12.968423 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:12.969196 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:12.969198 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:12.969745 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:12.969747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:12.970120 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:12.970121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:12.970181 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:12.970611 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:12.970616 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:12.972330 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
W20260924 15:19:12.972522 8654 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:12.967918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:55653 (local address 127.31.12.126:43993)
0924 15:19:12.967985 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:12.967986 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:12.968070 (+ 84us) server_negotiation.cc:407] Connection header received
0924 15:19:12.968127 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:12.968127 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:12.968138 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:12.968194 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:12.968505 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:12.969112 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:12.969829 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:12.970032 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:12.970120 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:12.972456 (+ 2336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.66:55653: BlockingRecv error: recv got EOF from 127.31.12.66:55653 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 15:19:13 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:13 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:13.099473 8656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:13.094853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.67:59489)
0924 15:19:13.095051 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:13.095062 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:13.095126 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:13.095241 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:13.095241 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:13.095400 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:13.095445 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:13.095447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:13.096070 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:13.096071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:13.096581 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:13.096582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:13.096905 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:13.096906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:13.096943 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:13.097363 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:13.097367 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:13.099418 (+ 2051us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":118,"threads_started":1}
W20260924 15:19:13.099583 8654 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:13.094918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:59489 (local address 127.31.12.126:43993)
0924 15:19:13.094989 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:13.094991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:13.095128 (+ 137us) server_negotiation.cc:407] Connection header received
0924 15:19:13.095181 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:13.095182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:13.095194 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:13.095236 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:13.095506 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:13.096003 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:13.096659 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:13.096841 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:13.096894 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:13.099512 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.67:59489: BlockingRecv error: recv got EOF from 127.31.12.67:59489 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 15:19:13 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:13 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:13.847831 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:13.842699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.65:47057)
0924 15:19:13.842863 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:13.842878 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:13.842924 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:13.843098 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:13.843098 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:13.843265 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:13.843334 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:13.843335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:13.844300 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:13.844302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:13.844826 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:13.844828 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:13.845177 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:13.845178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:13.845217 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:13.845663 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:13.845668 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:13.847730 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":106,"threads_started":1}
W20260924 15:19:13.847914 8658 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:13.842848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:47057 (local address 127.31.12.126:43993)
0924 15:19:13.842974 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:13.842976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:13.842984 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:19:13.843005 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:13.843006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:13.843016 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:13.843079 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:13.843481 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:13.844231 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:13.844913 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:13.845113 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:13.845164 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:13.847812 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.65:47057: BlockingRecv error: recv got EOF from 127.31.12.65:47057 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
Sep 24 15:19:13 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:13 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:13.978137 8659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:13.973116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.66:38583)
0924 15:19:13.973243 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:13.973256 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:13.973320 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:13.973572 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:13.973573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:13.973748 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:13.973796 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:13.973798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:13.974717 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:13.974718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:13.975269 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:13.975271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:13.975656 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:13.975657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:13.975715 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:13.976218 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:13.976224 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:13.978042 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
W20260924 15:19:13.978189 8658 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:13.973164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:38583 (local address 127.31.12.126:43993)
0924 15:19:13.973213 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:13.973215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:13.973324 (+ 109us) server_negotiation.cc:407] Connection header received
0924 15:19:13.973369 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:13.973370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:13.973385 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:13.973480 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:13.973879 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:13.974611 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:13.975353 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:13.975570 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:13.975658 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:13.978125 (+ 2467us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.66:38583: BlockingRecv error: recv got EOF from 127.31.12.66:38583 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 15:19:14 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:14 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:14.104727 8660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:14.099953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.67:55735)
0924 15:19:14.100117 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:14.100130 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:14.100222 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:14.100379 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:14.100380 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:14.100561 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:14.100606 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:14.100607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:14.101387 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:14.101388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:14.101950 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:14.101951 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:14.102327 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:14.102328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:14.102388 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:14.102885 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:14.102890 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:14.104627 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
W20260924 15:19:14.104780 8658 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:14.100042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:55735 (local address 127.31.12.126:43993)
0924 15:19:14.100114 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:14.100117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:14.100202 (+ 85us) server_negotiation.cc:407] Connection header received
0924 15:19:14.100288 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:14.100289 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:14.100309 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:14.100373 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:14.100686 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:14.101307 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:14.102048 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:14.102242 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:14.102327 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:14.104714 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.67:55735: BlockingRecv error: recv got EOF from 127.31.12.67:55735 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 15:19:14 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:14 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:14.853077 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:14.848318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.65:50455)
0924 15:19:14.848518 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:14.848531 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:14.848616 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:14.848824 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:14.848826 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:14.849072 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:14.849139 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:14.849142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:14.849806 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:14.849807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:14.850331 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:14.850333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:14.850703 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:14.850704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:14.850748 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:14.851183 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:14.851187 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:14.853018 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":92,"threads_started":1}
W20260924 15:19:14.853259 8662 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:14.848442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:50455 (local address 127.31.12.126:43993)
0924 15:19:14.848584 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:14.848586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:14.848597 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:19:14.848706 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:14.848707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:14.848723 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:14.848801 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:14.849221 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:14.849740 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:14.850423 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:14.850625 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:14.850687 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:14.853148 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.65:50455: BlockingRecv error: recv got EOF from 127.31.12.65:50455 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
Sep 24 15:19:14 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:14 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:14.983434 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:14.978617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.66:59421)
0924 15:19:14.978828 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:14.978839 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:14.978937 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:14.979088 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:14.979089 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:14.979276 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:14.979325 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:14.979327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:14.980010 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:14.980012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:14.980549 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:14.980550 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:14.980950 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:14.980951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:14.981013 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:14.981578 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:14.981583 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:14.983344 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":92,"threads_started":1}
W20260924 15:19:14.983495 8662 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:14.978704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:59421 (local address 127.31.12.126:43993)
0924 15:19:14.978803 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:14.978806 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:14.978915 (+ 109us) server_negotiation.cc:407] Connection header received
0924 15:19:14.978995 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:14.978996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:14.979010 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:14.979082 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:14.979422 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:14.979923 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:14.980637 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:14.980837 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:14.980936 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:14.983436 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.66:59421: BlockingRecv error: recv got EOF from 127.31.12.66:59421 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 15:19:15 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:15 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:15.110044 8664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:15.105316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.67:45951)
0924 15:19:15.105532 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:15.105543 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:15.105606 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:15.105778 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:15.105778 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:15.105955 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:15.106019 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:15.106020 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:15.106690 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:15.106692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:15.107256 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:15.107258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:15.107597 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:15.107598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:15.107637 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:15.108053 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:15.108057 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:15.109985 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":96,"threads_started":1}
W20260924 15:19:15.110208 8662 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:15.105375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:45951 (local address 127.31.12.126:43993)
0924 15:19:15.105463 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:15.105465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:15.105605 (+ 140us) server_negotiation.cc:407] Connection header received
0924 15:19:15.105683 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:15.105684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:15.105696 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:15.105768 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:15.106085 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:15.106595 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:15.107329 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:15.107529 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:15.107585 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:15.110103 (+ 2518us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.67:45951: BlockingRecv error: recv got EOF from 127.31.12.67:45951 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 24 15:19:15 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:15 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:15.861999 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:15.853597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.65:48837)
0924 15:19:15.853749 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:15.853760 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:15.853817 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:15.853951 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:15.853951 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:15.854111 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:15.854162 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:15.854164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:15.854875 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:15.854876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:15.855384 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:15.855385 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:15.855724 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:15.855725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:15.855766 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:15.859803 (+ 4037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:15.859808 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:15.861843 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
W20260924 15:19:15.862062 8666 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:15.853645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:48837 (local address 127.31.12.126:43993)
0924 15:19:15.853755 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:15.853757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:15.853822 (+ 65us) server_negotiation.cc:407] Connection header received
0924 15:19:15.853859 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:15.853860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:15.853873 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:15.853935 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:15.854230 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:15.854807 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:15.855455 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:15.855662 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:15.855716 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:15.861962 (+ 6246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.65:48837: BlockingRecv error: recv got EOF from 127.31.12.65:48837 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":56,"threads_started":1}
Sep 24 15:19:15 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:15 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:15.988505 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:15.983888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.66:35057)
0924 15:19:15.984062 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:15.984073 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:15.984156 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:15.984307 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:15.984307 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:15.984485 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:15.984542 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:15.984545 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:15.985221 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:15.985223 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:15.985820 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:15.985822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:15.986194 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:15.986195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:15.986258 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:15.986722 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:15.986726 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:15.988415 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":104,"threads_started":1}
W20260924 15:19:15.988593 8666 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:15.983979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:35057 (local address 127.31.12.126:43993)
0924 15:19:15.984054 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:15.984057 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:15.984140 (+ 83us) server_negotiation.cc:407] Connection header received
0924 15:19:15.984236 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:15.984237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:15.984249 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:15.984302 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:15.984623 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:15.985138 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:15.985904 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:15.986122 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:15.986197 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:15.988512 (+ 2315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.66:35057: BlockingRecv error: recv got EOF from 127.31.12.66:35057 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 24 15:19:16 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:16 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:16.115181 8668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:16.110596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.67:59789)
0924 15:19:16.110732 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:16.110746 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:16.110807 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:16.110951 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:16.110952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:16.111176 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:16.111224 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:16.111225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:16.111859 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:16.111860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:16.112396 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:16.112398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:16.112711 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:16.112712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:16.112755 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:16.113276 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:16.113281 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:16.115115 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
W20260924 15:19:16.115283 8666 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:16.110717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:59789 (local address 127.31.12.126:43993)
0924 15:19:16.110785 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:16.110787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:16.110795 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:19:16.110874 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:16.110875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:16.110891 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:16.110943 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:16.111285 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:16.111796 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:16.112462 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:16.112649 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:16.112696 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:16.115192 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.67:59789: BlockingRecv error: recv got EOF from 127.31.12.67:59789 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 24 15:19:16 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:16 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:16.866918 8669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:16.862474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.65:56279)
0924 15:19:16.862593 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:16.862604 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:16.862664 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:16.862800 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:16.862801 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:16.862997 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:16.863049 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:16.863051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:16.863686 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:16.863686 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:16.864214 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:16.864216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:16.864534 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:16.864535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:16.864575 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:16.865018 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:16.865023 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:16.866830 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
W20260924 15:19:16.867017 8670 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:16.862506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:56279 (local address 127.31.12.126:43993)
0924 15:19:16.862617 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:16.862619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:16.862657 (+ 38us) server_negotiation.cc:407] Connection header received
0924 15:19:16.862718 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:16.862719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:16.862729 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:16.862785 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:16.863112 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:16.863636 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:16.864288 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:16.864474 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:16.864521 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:16.866945 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.65:56279: BlockingRecv error: recv got EOF from 127.31.12.65:56279 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":49,"threads_started":1}
Sep 24 15:19:16 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:16 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:16.993762 8671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:16.989031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.66:46535)
0924 15:19:16.989181 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:16.989193 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:16.989261 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:16.989393 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:16.989394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:16.989583 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:16.989632 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:16.989633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:16.990302 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:16.990303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:16.990835 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:16.990837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:16.991209 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:16.991210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:16.991249 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:16.991775 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:16.991781 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:16.993647 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
W20260924 15:19:16.993804 8670 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:16.989123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:46535 (local address 127.31.12.126:43993)
0924 15:19:16.989191 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:16.989193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:16.989261 (+ 68us) server_negotiation.cc:407] Connection header received
0924 15:19:16.989333 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:16.989334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:16.989344 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:16.989388 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:16.989699 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:16.990215 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:16.990929 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:16.991139 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:16.991197 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:16.993740 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.66:46535: BlockingRecv error: recv got EOF from 127.31.12.66:46535 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 15:19:17 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:17 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:17.120700 8672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:17.115721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.67:50873)
0924 15:19:17.115969 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:17.115980 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:17.116039 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:17.116271 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:17.116272 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:17.116497 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:17.116546 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:17.116547 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:17.117243 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:17.117244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:17.117848 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:17.117850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:17.118231 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:17.118232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:17.118322 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:17.118759 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:17.118764 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:17.120595 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":102,"threads_started":1}
W20260924 15:19:17.120733 8670 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:17.115845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:50873 (local address 127.31.12.126:43993)
0924 15:19:17.115896 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:17.115898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:17.116030 (+ 132us) server_negotiation.cc:407] Connection header received
0924 15:19:17.116102 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:17.116103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:17.116117 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:17.116169 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:17.116651 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:17.117151 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:17.117954 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:17.118157 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:17.118212 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:17.120667 (+ 2455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.67:50873: BlockingRecv error: recv got EOF from 127.31.12.67:50873 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 15:19:17 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:17 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:17.871920 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:17.867444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.65:58065)
0924 15:19:17.867605 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:17.867616 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:17.867665 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:17.867826 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:17.867827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:17.867996 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:17.868044 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:17.868046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:17.868664 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:17.868664 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:17.869176 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:17.869177 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:17.869525 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:17.869525 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:17.869566 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:17.870103 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:17.870107 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:17.871861 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":97,"threads_started":1}
W20260924 15:19:17.872165 8674 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:17.867559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:58065 (local address 127.31.12.126:43993)
0924 15:19:17.867698 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:17.867699 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:17.867707 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:19:17.867730 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:17.867731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:17.867741 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:17.867808 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:17.868110 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:17.868600 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:17.869243 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:17.869426 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:17.869520 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:17.872025 (+ 2505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.65:58065: BlockingRecv error: recv got EOF from 127.31.12.65:58065 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
Sep 24 15:19:17 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:17 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:17.998729 8675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:17.994230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.66:36107)
0924 15:19:17.994402 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:17.994413 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:17.994497 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:17.994648 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:17.994648 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:17.994812 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:17.994860 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:17.994861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:17.995488 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:17.995489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:17.996015 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:17.996017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:17.996324 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:17.996324 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:17.996366 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:17.996796 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:17.996800 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:17.998631 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":87,"threads_started":1}
W20260924 15:19:17.998782 8674 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:17.994350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:36107 (local address 127.31.12.126:43993)
0924 15:19:17.994426 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:17.994428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:17.994531 (+ 103us) server_negotiation.cc:407] Connection header received
0924 15:19:17.994562 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:17.994562 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:17.994574 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:17.994630 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:17.994940 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:17.995425 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:17.996085 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:17.996263 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:17.996313 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:17.998720 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.66:36107: BlockingRecv error: recv got EOF from 127.31.12.66:36107 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 24 15:19:18 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:18 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:18.126036 8676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:18.121242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.67:36067)
0924 15:19:18.121371 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:18.121388 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:18.121453 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:18.121682 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:18.121683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:18.121889 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:18.121936 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:18.121938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:18.122641 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:18.122643 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:18.123204 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:18.123207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:18.123582 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:18.123584 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:18.123657 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:18.124102 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:18.124106 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:18.125967 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":74,"threads_started":1}
W20260924 15:19:18.126169 8674 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:18.121476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:36067 (local address 127.31.12.126:43993)
0924 15:19:18.121553 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:18.121555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:18.121562 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:19:18.121581 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:18.121582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:18.121593 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:18.121672 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:18.122024 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:18.122537 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:18.123293 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:18.123494 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:18.123580 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:18.126084 (+ 2504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.67:36067: BlockingRecv error: recv got EOF from 127.31.12.67:36067 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 24 15:19:18 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:18 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:18.876834 8677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:18.872385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.65:39767)
0924 15:19:18.872522 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:18.872541 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:18.872600 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:18.872729 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:18.872730 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:18.872897 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:18.872946 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:18.872948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:18.873594 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:18.873595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:18.874124 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:18.874125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:18.874453 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:18.874454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:18.874493 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:18.874961 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:18.874966 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:18.876768 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
W20260924 15:19:18.876986 8678 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:18.872462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:39767 (local address 127.31.12.126:43993)
0924 15:19:18.872591 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:18.872593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:18.872603 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:19:18.872636 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:18.872637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:18.872647 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:18.872709 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:18.873009 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:18.873530 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:18.874193 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:18.874389 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:18.874438 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:18.876857 (+ 2419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.65:39767: BlockingRecv error: recv got EOF from 127.31.12.65:39767 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
Sep 24 15:19:19 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:19 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:19.004014 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:18.999218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.66:51301)
0924 15:19:18.999371 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:18.999383 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:18.999470 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:18.999624 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:18.999625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:18.999823 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:18.999867 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:18.999869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:19.000567 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:19.000568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:19.001111 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:19.001113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:19.001553 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:19.001554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:19.001613 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:19.002095 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:19.002100 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:19.003905 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
W20260924 15:19:19.004065 8678 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:18.999361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:51301 (local address 127.31.12.126:43993)
0924 15:19:18.999414 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:18.999417 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:18.999450 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:19:18.999529 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:18.999530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:18.999545 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:18.999618 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:18.999955 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:19.000466 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:19.001220 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:19.001430 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:19.001537 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:19.003997 (+ 2460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.66:51301: BlockingRecv error: recv got EOF from 127.31.12.66:51301 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 24 15:19:19 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:19 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:19.131052 8680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:19.126507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.67:52621)
0924 15:19:19.126672 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:19.126683 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:19.126749 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:19.126874 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:19.126875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:19.127035 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:19.127082 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:19.127083 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:19.127771 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:19.127772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:19.128485 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:19.128488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:19.128790 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:19.128791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:19.128828 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:19.129276 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:19.129280 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:19.130996 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
W20260924 15:19:19.131148 8678 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:19.126618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:52621 (local address 127.31.12.126:43993)
0924 15:19:19.126666 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:19.126668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:19.126750 (+ 82us) server_negotiation.cc:407] Connection header received
0924 15:19:19.126792 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:19.126793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:19.126805 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:19.126853 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:19.127141 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:19.127688 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:19.128541 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:19.128722 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:19.128791 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:19.131081 (+ 2290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.67:52621: BlockingRecv error: recv got EOF from 127.31.12.67:52621 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 15:19:19 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:19 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:19.882074 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:19.877352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.65:60205)
0924 15:19:19.877525 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:19.877536 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:19.877604 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:19.877781 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:19.877782 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:19.878030 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:19.878091 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:19.878094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:19.878772 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:19.878773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:19.879319 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:19.879320 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:19.879641 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:19.879642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:19.879682 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:19.880134 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:19.880138 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:19.882003 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":86,"threads_started":1}
W20260924 15:19:19.882148 8682 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:19.877499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:60205 (local address 127.31.12.126:43993)
0924 15:19:19.877621 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:19.877623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:19.877633 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:19:19.877664 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:19.877664 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:19.877676 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:19.877759 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:19.878181 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:19.878693 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:19.879392 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:19.879579 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:19.879627 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:19.882092 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.65:60205: BlockingRecv error: recv got EOF from 127.31.12.65:60205 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
Sep 24 15:19:20 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:20 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:20.009353 8683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:20.004574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.66:48859)
0924 15:19:20.004745 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:20.004758 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:20.004866 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:20.005032 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:20.005033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:20.005221 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:20.005265 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:20.005267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:20.006053 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:20.006054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:20.006588 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:20.006590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:20.007004 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:20.007005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:20.007062 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:20.007567 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:20.007572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:20.009242 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":85,"threads_started":1}
W20260924 15:19:20.009409 8682 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:20.004674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:48859 (local address 127.31.12.126:43993)
0924 15:19:20.004747 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:20.004749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:20.004836 (+ 87us) server_negotiation.cc:407] Connection header received
0924 15:19:20.004939 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:20.004940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:20.004959 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:20.005026 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:20.005346 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:20.005969 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:20.006676 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:20.006929 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:20.007005 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:20.009345 (+ 2340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.66:48859: BlockingRecv error: recv got EOF from 127.31.12.66:48859 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 15:19:20 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:20 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.67@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:20.136014 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:20.131585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.67:33773)
0924 15:19:20.131726 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:20.131736 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:20.131796 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:20.131939 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:20.131939 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:20.132102 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:20.132150 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:20.132152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:20.132789 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:20.132791 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:20.133328 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:20.133330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:20.133730 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:20.133731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:20.133776 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:20.134196 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:20.134201 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:20.135917 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
W20260924 15:19:20.136067 8682 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:20.131656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:33773 (local address 127.31.12.126:43993)
0924 15:19:20.131747 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:20.131749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:20.131785 (+ 36us) server_negotiation.cc:407] Connection header received
0924 15:19:20.131857 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:20.131858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:20.131871 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:20.131915 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:20.132216 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:20.132699 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:20.133413 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:20.133660 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:20.133717 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:20.136002 (+ 2285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.67:33773: BlockingRecv error: recv got EOF from 127.31.12.67:33773 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 24 15:19:20 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:20 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.65@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:20.890620 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:20.882622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.65:36787)
0924 15:19:20.882765 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:20.882776 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:20.882824 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:20.882965 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:20.882966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:20.883128 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:20.883180 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:20.883181 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:20.883807 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:20.883808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:20.884573 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:20.884577 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:20.885011 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:20.885012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:20.885061 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:20.888828 (+ 3767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:20.888833 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:20.890562 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
W20260924 15:19:20.890748 8686 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:20.882712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:36787 (local address 127.31.12.126:43993)
0924 15:19:20.882848 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:20.882850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:20.882857 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:19:20.882879 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:20.882879 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:20.882889 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:20.882949 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:20.883243 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:20.883746 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:20.884652 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:20.884920 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:20.884976 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:20.890661 (+ 5685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.65:36787: BlockingRecv error: recv got EOF from 127.31.12.65:36787 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
Sep 24 15:19:21 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
Sep 24 15:19:21 dist-test-slave-zwbb krb5kdc[7614](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.31.12.66@KRBTEST.COM for oryx/127.31.12.126@KRBTEST.COM, Server not found in Kerberos database
I20260924 15:19:21.014770 8687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:21.010024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:43993 (local address 127.31.12.66:59149)
0924 15:19:21.010151 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:21.010161 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:21.010211 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:21.010426 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:21.010427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:21.010605 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:21.010652 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:21.010653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:21.011311 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:21.011312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:21.011857 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:21.011858 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:21.012217 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:21.012218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:21.012257 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:21.012689 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:21.012693 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:21.014692 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.31.12.126:43993: Server oryx/127.31.12.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
W20260924 15:19:21.014909 8686 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:21.010111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:59149 (local address 127.31.12.126:43993)
0924 15:19:21.010285 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:21.010287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:21.010297 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:19:21.010335 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:21.010335 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:21.010346 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:21.010409 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:21.010729 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:21.011238 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:21.011946 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:21.012150 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:21.012206 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:21.014811 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.31.12.66:59149: BlockingRecv error: recv got EOF from 127.31.12.66:59149 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
I20260924 15:19:21.057926 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 8204
I20260924 15:19:21.063499 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 8358
I20260924 15:19:21.067924 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 8491
I20260924 15:19:21.072000 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 8133
2026-09-24T15:19:21Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17489 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T15:19:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:19:21Z Disabled control of system clock
I20260924 15:19:21.100003 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:34439
--webserver_interface=127.31.12.126
--webserver_port=0
--builtin_ntp_servers=127.31.12.84:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:34439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 15:19:21.179796 8691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:21.180038 8691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:21.180058 8691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:21.181167 8691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:19:21.181201 8691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:21.181213 8691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:21.181227 8691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:19:21.181239 8691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:19:21.182502 8691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:34439
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:34439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8691
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:21.182686 8691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:21.182912 8691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:21.187991 8697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:21.187997 8696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:21.188051 8691 server_base.cc:1034] running on GCE node
W20260924 15:19:21.188051 8699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:21.188468 8691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:21.188710 8691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:21.189845 8691 hybrid_clock.cc:648] HybridClock initialized: now 1790263161189826 us; error 39 us; skew 500 ppm
I20260924 15:19:21.190966 8691 webserver.cc:460] Webserver started at http://127.31.12.126:37501/ using document root <none> and password file <none>
I20260924 15:19:21.191171 8691 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:21.191211 8691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:21.191299 8691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:19:21.192294 8691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "96cb8f13859d4053b0964c0bf94feb51"
format_stamp: "Formatted at 2026-09-24 15:19:21 on dist-test-slave-zwbb"
server_key: "f7777a88634fce3f773dc0dbe3b7185e"
server_key_iv: "390058eac74953b019b0a67e01c36de4"
server_key_version: "encryptionkey@0"
I20260924 15:19:21.192593 8691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "96cb8f13859d4053b0964c0bf94feb51"
format_stamp: "Formatted at 2026-09-24 15:19:21 on dist-test-slave-zwbb"
server_key: "f7777a88634fce3f773dc0dbe3b7185e"
server_key_iv: "390058eac74953b019b0a67e01c36de4"
server_key_version: "encryptionkey@0"
I20260924 15:19:21.193912 8691 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:19:21.194592 8705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:21.194775 8691 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:19:21.194840 8691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "96cb8f13859d4053b0964c0bf94feb51"
format_stamp: "Formatted at 2026-09-24 15:19:21 on dist-test-slave-zwbb"
server_key: "f7777a88634fce3f773dc0dbe3b7185e"
server_key_iv: "390058eac74953b019b0a67e01c36de4"
server_key_version: "encryptionkey@0"
I20260924 15:19:21.194911 8691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:21.208781 8691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:21.209034 8691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:21.209129 8691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:21.212931 8691 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:34439
I20260924 15:19:21.212966 8757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:34439 every 8 connection(s)
I20260924 15:19:21.213310 8691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:19:21.214067 8758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:21.214231 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 8691
I20260924 15:19:21.214329 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:19:21.214568 31793 external_mini_cluster.cc:1428] Setting key dd5d50a24965e4155d17eaf1c99d3274
I20260924 15:19:21.217552 8758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51: Bootstrap starting.
I20260924 15:19:21.218184 8758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:21.218482 8758 log.cc:824] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51: Log is configured to *not* fsync() on all Append() calls
I20260924 15:19:21.219201 8758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51: No bootstrap required, opened a new log
I20260924 15:19:21.220233 8761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:21.215118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39196 (local address 127.31.12.126:34439)
0924 15:19:21.215320 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:21.215335 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:21.215359 (+ 24us) server_negotiation.cc:407] Connection header received
0924 15:19:21.216097 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:21.216104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:21.216165 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:21.216328 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:21.216564 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:21.217416 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:21.218285 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:21.218731 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:21.218863 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:21.219561 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:21.219565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:21.219565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:19:21.219736 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:21.219772 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:19:21.219782 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:21.219859 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:21.219933 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 15:19:21.220019 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260924 15:19:21.221267 8758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96cb8f13859d4053b0964c0bf94feb51" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 34439 } }
I20260924 15:19:21.221421 8758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:21.221496 8758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96cb8f13859d4053b0964c0bf94feb51, State: Initialized, Role: FOLLOWER
I20260924 15:19:21.221625 8758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96cb8f13859d4053b0964c0bf94feb51" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 34439 } }
I20260924 15:19:21.221724 8758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:19:21.221769 8758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:19:21.221817 8758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:21.222326 8758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96cb8f13859d4053b0964c0bf94feb51" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 34439 } }
I20260924 15:19:21.222458 8758 leader_election.cc:302] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96cb8f13859d4053b0964c0bf94feb51; no voters:
I20260924 15:19:21.222635 8758 leader_election.cc:288] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:19:21.222685 8763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:19:21.222847 8763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51 [term 1 LEADER]: Becoming Leader. State: Replica: 96cb8f13859d4053b0964c0bf94feb51, State: Running, Role: LEADER
I20260924 15:19:21.222898 8758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:19:21.222982 8763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96cb8f13859d4053b0964c0bf94feb51" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 34439 } }
I20260924 15:19:21.223364 8764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96cb8f13859d4053b0964c0bf94feb51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96cb8f13859d4053b0964c0bf94feb51" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 34439 } } }
I20260924 15:19:21.223384 8765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96cb8f13859d4053b0964c0bf94feb51. Latest consensus state: current_term: 1 leader_uuid: "96cb8f13859d4053b0964c0bf94feb51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96cb8f13859d4053b0964c0bf94feb51" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 34439 } } }
I20260924 15:19:21.223498 8764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51 [sys.catalog]: This master's current role is: LEADER
I20260924 15:19:21.223536 8765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51 [sys.catalog]: This master's current role is: LEADER
W20260924 15:19:21.224682 8780 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:19:21.224746 8780 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:19:21.224804 8772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:19:21.225024 8772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:19:21.226495 8772 catalog_manager.cc:1347] Generated new cluster ID: 76a1b08597344729ae9bdcd35e9c49e7
I20260924 15:19:21.226543 8772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:19:21.245083 8772 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:19:21.245798 8772 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:19:21.259931 8772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96cb8f13859d4053b0964c0bf94feb51: Generated new TSK 0
I20260924 15:19:21.260136 8772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:19:21.265789 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:34439
--builtin_ntp_servers=127.31.12.84:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 15:19:21.346611 8783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:21.346804 8783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:21.346825 8783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:21.347952 8783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:21.347992 8783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:21.348016 8783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:19:21.349328 8783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:34439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:21.349589 8783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:21.349864 8783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:21.354842 8791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:21.354835 8789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:21.354859 8783 server_base.cc:1034] running on GCE node
W20260924 15:19:21.354859 8788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:21.355288 8783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:21.355530 8783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:21.356684 8783 hybrid_clock.cc:648] HybridClock initialized: now 1790263161356665 us; error 38 us; skew 500 ppm
I20260924 15:19:21.357961 8783 webserver.cc:460] Webserver started at http://127.31.12.65:41785/ using document root <none> and password file <none>
I20260924 15:19:21.358217 8783 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:21.358287 8783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:21.358438 8783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:19:21.359486 8783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "f581235d17474caa9731d28c985baa8e"
format_stamp: "Formatted at 2026-09-24 15:19:21 on dist-test-slave-zwbb"
server_key: "40ebaf76303759e9848f488ac2931cce"
server_key_iv: "60e92323f347ab299e606c60f7adacea"
server_key_version: "encryptionkey@0"
I20260924 15:19:21.359856 8783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "f581235d17474caa9731d28c985baa8e"
format_stamp: "Formatted at 2026-09-24 15:19:21 on dist-test-slave-zwbb"
server_key: "40ebaf76303759e9848f488ac2931cce"
server_key_iv: "60e92323f347ab299e606c60f7adacea"
server_key_version: "encryptionkey@0"
I20260924 15:19:21.361270 8783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:19:21.362150 8797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:21.362319 8783 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:19:21.362412 8783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "f581235d17474caa9731d28c985baa8e"
format_stamp: "Formatted at 2026-09-24 15:19:21 on dist-test-slave-zwbb"
server_key: "40ebaf76303759e9848f488ac2931cce"
server_key_iv: "60e92323f347ab299e606c60f7adacea"
server_key_version: "encryptionkey@0"
I20260924 15:19:21.362506 8783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:21.374719 8783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:21.375015 8783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:21.375167 8783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:21.375379 8783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:19:21.375696 8783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:19:21.375751 8783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:21.375789 8783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:19:21.375818 8783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:21.381657 8783 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:36497
I20260924 15:19:21.381716 8911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:36497 every 8 connection(s)
I20260924 15:19:21.382023 8783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:19:21.387130 8761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:21.382819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:47163 (local address 127.31.12.126:34439)
0924 15:19:21.382918 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:21.382920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:21.383171 (+ 251us) server_negotiation.cc:407] Connection header received
0924 15:19:21.383695 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:21.383696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:21.383711 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:21.383763 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:21.384406 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:21.384905 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:21.385931 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:21.386130 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:21.386184 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:21.386890 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:21.386891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:21.386891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:19:21.386895 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:21.386909 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:19:21.386912 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:21.386954 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:21.387050 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 15:19:21.387072 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 15:19:21.387342 8914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:21.382776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:34439 (local address 127.31.12.65:47163)
0924 15:19:21.383044 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:21.383091 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:21.383602 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:21.383821 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:21.383824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:21.383832 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:21.384325 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:21.384330 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:21.385006 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:21.385007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:21.385859 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:21.385862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:21.386207 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:21.386207 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:21.386309 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:21.386769 (+ 460us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:19:21.386769 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:21.386795 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:19:21.386816 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:19:21.386980 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:19:21.386984 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:19:21.386986 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:19:21.387039 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 15:19:21.387113 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":112,"threads_started":1}
I20260924 15:19:21.387879 8912 heartbeater.cc:347] Connected to a master server at 127.31.12.126:34439
I20260924 15:19:21.387992 8912 heartbeater.cc:464] Registering TS with master...
I20260924 15:19:21.388163 8912 heartbeater.cc:511] Master 127.31.12.126:34439 requested a full tablet report, sending...
I20260924 15:19:21.388648 8722 ts_manager.cc:194] Registered new tserver with Master: f581235d17474caa9731d28c985baa8e (127.31.12.65:36497)
I20260924 15:19:21.389204 8722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.12.65:47163
I20260924 15:19:21.390707 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 8783
I20260924 15:19:21.390808 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:19:21.390998 31793 external_mini_cluster.cc:1428] Setting key 6ac1855c1a1d73c3aea562a0e8b936e4
I20260924 15:19:21.391848 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:34439
--builtin_ntp_servers=127.31.12.84:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 15:19:21.471288 8915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:21.471467 8915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:21.471495 8915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:21.472635 8915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:21.472674 8915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:21.472698 8915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:19:21.474030 8915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:34439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:21.474262 8915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:21.474558 8915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:21.479394 8923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:21.479363 8921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:21.479378 8920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:21.479534 8915 server_base.cc:1034] running on GCE node
I20260924 15:19:21.479884 8915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:21.480136 8915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:21.481295 8915 hybrid_clock.cc:648] HybridClock initialized: now 1790263161481284 us; error 43 us; skew 500 ppm
I20260924 15:19:21.482556 8915 webserver.cc:460] Webserver started at http://127.31.12.66:45163/ using document root <none> and password file <none>
I20260924 15:19:21.482810 8915 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:21.482875 8915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:21.483023 8915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:19:21.484027 8915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "8ce2853aee75422783652743d7bae7db"
format_stamp: "Formatted at 2026-09-24 15:19:21 on dist-test-slave-zwbb"
server_key: "81e053f85f8a4f577c16e0089a583117"
server_key_iv: "58261412cd24df207b3143f629374f96"
server_key_version: "encryptionkey@0"
I20260924 15:19:21.484371 8915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "8ce2853aee75422783652743d7bae7db"
format_stamp: "Formatted at 2026-09-24 15:19:21 on dist-test-slave-zwbb"
server_key: "81e053f85f8a4f577c16e0089a583117"
server_key_iv: "58261412cd24df207b3143f629374f96"
server_key_version: "encryptionkey@0"
I20260924 15:19:21.485687 8915 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:19:21.486418 8929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:21.486581 8915 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:19:21.486688 8915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "8ce2853aee75422783652743d7bae7db"
format_stamp: "Formatted at 2026-09-24 15:19:21 on dist-test-slave-zwbb"
server_key: "81e053f85f8a4f577c16e0089a583117"
server_key_iv: "58261412cd24df207b3143f629374f96"
server_key_version: "encryptionkey@0"
I20260924 15:19:21.486760 8915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:21.498008 8915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:21.498261 8915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:21.498404 8915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:21.498613 8915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:19:21.498910 8915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:19:21.498972 8915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:21.499012 8915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:19:21.499043 8915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:21.504534 8915 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:38517
I20260924 15:19:21.504607 9042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:38517 every 8 connection(s)
I20260924 15:19:21.504935 8915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:19:21.506570 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 8915
I20260924 15:19:21.506662 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:19:21.506904 31793 external_mini_cluster.cc:1428] Setting key abca79d275a0657d563cca22b0721b3d
I20260924 15:19:21.507866 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:34439
--builtin_ntp_servers=127.31.12.84:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 15:19:21.510288 8761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:21.505685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:49007 (local address 127.31.12.126:34439)
0924 15:19:21.505731 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:21.505733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:21.505945 (+ 212us) server_negotiation.cc:407] Connection header received
0924 15:19:21.506414 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:21.506415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:21.506425 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:21.506482 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:21.507429 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:21.507929 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:21.508862 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:21.509057 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:21.509135 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:21.510001 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:21.510002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:21.510003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:19:21.510008 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:21.510023 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:19:21.510027 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:21.510095 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:21.510185 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 15:19:21.510212 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:19:21.510489 9045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:21.505673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:34439 (local address 127.31.12.66:49007)
0924 15:19:21.505834 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:21.505864 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:21.506312 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:21.506560 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:21.506564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:21.506574 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:21.507330 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:21.507335 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:21.508025 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:21.508026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:21.508762 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:21.508764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:21.509147 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:21.509148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:21.509276 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:21.509840 (+ 564us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:19:21.509841 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:21.509865 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:19:21.509886 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:19:21.510095 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:19:21.510101 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:19:21.510104 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:19:21.510187 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 15:19:21.510284 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260924 15:19:21.510895 9043 heartbeater.cc:347] Connected to a master server at 127.31.12.126:34439
I20260924 15:19:21.510982 9043 heartbeater.cc:464] Registering TS with master...
I20260924 15:19:21.511162 9043 heartbeater.cc:511] Master 127.31.12.126:34439 requested a full tablet report, sending...
I20260924 15:19:21.511523 8722 ts_manager.cc:194] Registered new tserver with Master: 8ce2853aee75422783652743d7bae7db (127.31.12.66:38517)
I20260924 15:19:21.511880 8722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.12.66:49007
W20260924 15:19:21.588203 9046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:21.588429 9046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:21.588474 9046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:21.589690 9046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:21.589751 9046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:21.589793 9046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:19:21.591171 9046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:34439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:21.591434 9046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:21.591756 9046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:21.596668 9051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:21.596675 9054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:21.596675 9052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:21.596956 9046 server_base.cc:1034] running on GCE node
I20260924 15:19:21.597203 9046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:21.597483 9046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:21.598631 9046 hybrid_clock.cc:648] HybridClock initialized: now 1790263161598600 us; error 46 us; skew 500 ppm
I20260924 15:19:21.599817 9046 webserver.cc:460] Webserver started at http://127.31.12.67:34345/ using document root <none> and password file <none>
I20260924 15:19:21.600085 9046 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:21.600162 9046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:21.600317 9046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:19:21.601346 9046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "6e5dc5d0c1a34551b81037ac89bc23b0"
format_stamp: "Formatted at 2026-09-24 15:19:21 on dist-test-slave-zwbb"
server_key: "54440f7b4e5b231cbe8bfd874a9529f8"
server_key_iv: "3e037a33ca1373486c70c01b184ad865"
server_key_version: "encryptionkey@0"
I20260924 15:19:21.601769 9046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "6e5dc5d0c1a34551b81037ac89bc23b0"
format_stamp: "Formatted at 2026-09-24 15:19:21 on dist-test-slave-zwbb"
server_key: "54440f7b4e5b231cbe8bfd874a9529f8"
server_key_iv: "3e037a33ca1373486c70c01b184ad865"
server_key_version: "encryptionkey@0"
I20260924 15:19:21.603039 9046 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:19:21.603734 9060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:21.603909 9046 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:19:21.604009 9046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "6e5dc5d0c1a34551b81037ac89bc23b0"
format_stamp: "Formatted at 2026-09-24 15:19:21 on dist-test-slave-zwbb"
server_key: "54440f7b4e5b231cbe8bfd874a9529f8"
server_key_iv: "3e037a33ca1373486c70c01b184ad865"
server_key_version: "encryptionkey@0"
I20260924 15:19:21.604096 9046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:21.625425 9046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:21.625780 9046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:21.625957 9046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:21.626210 9046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:19:21.626570 9046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:19:21.626631 9046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:21.626677 9046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:19:21.626709 9046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:21.632741 9046 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:43981
I20260924 15:19:21.632814 9173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:43981 every 8 connection(s)
I20260924 15:19:21.633193 9046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:19:21.638412 8761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:21.633985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:36637 (local address 127.31.12.126:34439)
0924 15:19:21.634061 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:21.634062 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:21.634304 (+ 242us) server_negotiation.cc:407] Connection header received
0924 15:19:21.634818 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:21.634819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:21.634829 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:21.634882 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:21.635562 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:21.636042 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:21.637056 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:21.637232 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:21.637280 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:21.638147 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:21.638148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:21.638149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:19:21.638153 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:21.638166 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:19:21.638169 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:21.638209 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:21.638335 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 15:19:21.638355 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 15:19:21.638648 9176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:21.634002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:34439 (local address 127.31.12.67:36637)
0924 15:19:21.634179 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:21.634214 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:21.634727 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:21.634942 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:21.634946 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:21.634954 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:21.635490 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:21.635495 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:21.636127 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:21.636128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:21.636979 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:21.636982 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:21.637310 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:21.637311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:21.637483 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:21.638018 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:19:21.638018 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:21.638047 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:19:21.638069 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:19:21.638248 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:19:21.638254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:19:21.638257 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:19:21.638326 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 15:19:21.638404 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260924 15:19:21.639202 9174 heartbeater.cc:347] Connected to a master server at 127.31.12.126:34439
I20260924 15:19:21.639277 9174 heartbeater.cc:464] Registering TS with master...
I20260924 15:19:21.639475 9174 heartbeater.cc:511] Master 127.31.12.126:34439 requested a full tablet report, sending...
I20260924 15:19:21.639847 8722 ts_manager.cc:194] Registered new tserver with Master: 6e5dc5d0c1a34551b81037ac89bc23b0 (127.31.12.67:43981)
I20260924 15:19:21.640165 8722 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.12.67:36637
I20260924 15:19:21.643173 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 9046
I20260924 15:19:21.643249 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:19:21.643399 31793 external_mini_cluster.cc:1428] Setting key 7e6e25516471093694a1d7ad60bf03d2
I20260924 15:19:21.644171 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 15:19:21.645952 8761 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:21.645425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39202 (local address 127.31.12.126:34439)
0924 15:19:21.645540 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:21.645542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:21.645641 (+ 99us) server_negotiation.cc:407] Connection header received
0924 15:19:21.645708 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:21.645709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:21.645719 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:21.645773 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:21.645893 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39202: BlockingRecv error: recv got EOF from 127.0.0.1:39202 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 15:19:21.646044 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 8783
I20260924 15:19:21.652981 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 8915
I20260924 15:19:21.657542 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 9046
I20260924 15:19:21.661651 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 8691
2026-09-24T15:19:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (588 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T15:19:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:19:21Z Disabled control of system clock
I20260924 15:19:21.688241 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:44443
--webserver_interface=127.31.12.126
--webserver_port=0
--builtin_ntp_servers=127.31.12.84:42135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:44443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 15:19:21.768139 9185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:21.768381 9185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:21.768416 9185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:21.769582 9185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:19:21.769650 9185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:21.769672 9185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:21.769691 9185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:19:21.769724 9185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:19:21.771081 9185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:42135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:44443
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:44443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:21.771353 9185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:21.771636 9185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:21.776327 9190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:21.776341 9193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:21.776396 9191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:21.776604 9185 server_base.cc:1034] running on GCE node
I20260924 15:19:21.776846 9185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:21.777136 9185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:21.778306 9185 hybrid_clock.cc:648] HybridClock initialized: now 1790263161778298 us; error 34 us; skew 500 ppm
I20260924 15:19:21.779452 9185 webserver.cc:460] Webserver started at http://127.31.12.126:35439/ using document root <none> and password file <none>
I20260924 15:19:21.779709 9185 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:21.779781 9185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:21.779934 9185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:19:21.780985 9185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "33b5942b845945ada1e2a72e7d371199"
format_stamp: "Formatted at 2026-09-24 15:19:21 on dist-test-slave-zwbb"
server_key: "4ab4e1bd6cd636d5cfa4b70a39284749"
server_key_iv: "742e3c05959c8bdb8a5af675af43b240"
server_key_version: "encryptionkey@0"
I20260924 15:19:21.781373 9185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "33b5942b845945ada1e2a72e7d371199"
format_stamp: "Formatted at 2026-09-24 15:19:21 on dist-test-slave-zwbb"
server_key: "4ab4e1bd6cd636d5cfa4b70a39284749"
server_key_iv: "742e3c05959c8bdb8a5af675af43b240"
server_key_version: "encryptionkey@0"
I20260924 15:19:21.782780 9185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:19:21.783517 9199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:21.783706 9185 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:19:21.783811 9185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "33b5942b845945ada1e2a72e7d371199"
format_stamp: "Formatted at 2026-09-24 15:19:21 on dist-test-slave-zwbb"
server_key: "4ab4e1bd6cd636d5cfa4b70a39284749"
server_key_iv: "742e3c05959c8bdb8a5af675af43b240"
server_key_version: "encryptionkey@0"
I20260924 15:19:21.783885 9185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:21.787247 9185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:21.787509 9185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:21.787649 9185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:21.791404 9185 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:44443
I20260924 15:19:21.791436 9251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:44443 every 8 connection(s)
I20260924 15:19:21.791751 9185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:19:21.792274 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 9185
I20260924 15:19:21.792353 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:19:21.792387 9252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:21.792551 31793 external_mini_cluster.cc:1428] Setting key 609ecb9746fc1cffe58e9d2013026d63
I20260924 15:19:21.795399 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:21.793111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.31.12.126:44443)
0924 15:19:21.793340 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:21.793350 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:21.793392 (+ 42us) server_negotiation.cc:407] Connection header received
0924 15:19:21.794142 (+ 750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:21.794151 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:21.794205 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:21.794358 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:21.794996 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:21.795000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:21.795000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:19:21.795005 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:21.795049 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:19:21.795062 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:21.795140 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:21.795174 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 15:19:21.795227 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260924 15:19:21.796420 9252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199: Bootstrap starting.
I20260924 15:19:21.796999 9252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:21.797315 9252 log.cc:824] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199: Log is configured to *not* fsync() on all Append() calls
I20260924 15:19:21.798012 9252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199: No bootstrap required, opened a new log
I20260924 15:19:21.799259 9252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b5942b845945ada1e2a72e7d371199" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44443 } }
I20260924 15:19:21.799384 9252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:21.799407 9252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33b5942b845945ada1e2a72e7d371199, State: Initialized, Role: FOLLOWER
I20260924 15:19:21.799525 9252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b5942b845945ada1e2a72e7d371199" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44443 } }
I20260924 15:19:21.799600 9252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:19:21.799623 9252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:19:21.799649 9252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:21.800094 9252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b5942b845945ada1e2a72e7d371199" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44443 } }
I20260924 15:19:21.800175 9252 leader_election.cc:302] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33b5942b845945ada1e2a72e7d371199; no voters:
I20260924 15:19:21.800315 9252 leader_election.cc:288] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:19:21.800585 9257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:19:21.800609 9252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:19:21.800712 9257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199 [term 1 LEADER]: Becoming Leader. State: Replica: 33b5942b845945ada1e2a72e7d371199, State: Running, Role: LEADER
I20260924 15:19:21.800822 9257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b5942b845945ada1e2a72e7d371199" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44443 } }
I20260924 15:19:21.801205 9259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33b5942b845945ada1e2a72e7d371199. Latest consensus state: current_term: 1 leader_uuid: "33b5942b845945ada1e2a72e7d371199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b5942b845945ada1e2a72e7d371199" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44443 } } }
I20260924 15:19:21.801206 9258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33b5942b845945ada1e2a72e7d371199" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33b5942b845945ada1e2a72e7d371199" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44443 } } }
I20260924 15:19:21.801366 9259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199 [sys.catalog]: This master's current role is: LEADER
I20260924 15:19:21.801384 9258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199 [sys.catalog]: This master's current role is: LEADER
I20260924 15:19:21.801674 9267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:19:21.802260 9267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:19:21.803771 9267 catalog_manager.cc:1347] Generated new cluster ID: 0002499d899b45d0bf2a53ed1bcfd651
I20260924 15:19:21.803823 9267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:19:21.827145 9267 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:19:21.827368 9267 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:19:21.835438 9267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33b5942b845945ada1e2a72e7d371199: Generated new TSK 0
I20260924 15:19:21.835659 9267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:19:21.845733 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:44443
--builtin_ntp_servers=127.31.12.84:42135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 15:19:21.940232 9277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:21.940441 9277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:21.940462 9277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:21.941694 9277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:21.941731 9277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:21.941756 9277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:19:21.943080 9277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:42135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:44443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:21.943307 9277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:21.943621 9277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:21.949357 9282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:21.949568 9277 server_base.cc:1034] running on GCE node
W20260924 15:19:21.949602 9285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:21.949394 9283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:21.949811 9277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:21.950059 9277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:21.951212 9277 hybrid_clock.cc:648] HybridClock initialized: now 1790263161951197 us; error 30 us; skew 500 ppm
I20260924 15:19:21.952453 9277 webserver.cc:460] Webserver started at http://127.31.12.65:44983/ using document root <none> and password file <none>
I20260924 15:19:21.952702 9277 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:21.952769 9277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:21.952919 9277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:19:21.954044 9277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "9c8a1b0a76e74047b441bb380f2e58e2"
format_stamp: "Formatted at 2026-09-24 15:19:21 on dist-test-slave-zwbb"
server_key: "b2c4ae8daa281ab6ac2e2f837a58cb7b"
server_key_iv: "76c6d1511df0a1a3f190e8a7317743e1"
server_key_version: "encryptionkey@0"
I20260924 15:19:21.954398 9277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "9c8a1b0a76e74047b441bb380f2e58e2"
format_stamp: "Formatted at 2026-09-24 15:19:21 on dist-test-slave-zwbb"
server_key: "b2c4ae8daa281ab6ac2e2f837a58cb7b"
server_key_iv: "76c6d1511df0a1a3f190e8a7317743e1"
server_key_version: "encryptionkey@0"
I20260924 15:19:21.955703 9277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:19:21.956476 9291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:21.956650 9277 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:19:21.956748 9277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "9c8a1b0a76e74047b441bb380f2e58e2"
format_stamp: "Formatted at 2026-09-24 15:19:21 on dist-test-slave-zwbb"
server_key: "b2c4ae8daa281ab6ac2e2f837a58cb7b"
server_key_iv: "76c6d1511df0a1a3f190e8a7317743e1"
server_key_version: "encryptionkey@0"
I20260924 15:19:21.956827 9277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:21.960011 9277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:21.960261 9277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:21.960397 9277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:21.960615 9277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:19:21.960909 9277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:19:21.960994 9277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:21.961026 9277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:19:21.961052 9277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:21.966517 9277 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:43433
I20260924 15:19:21.966609 9404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:43433 every 8 connection(s)
I20260924 15:19:21.966878 9277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:19:21.969911 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:21.967615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:50427 (local address 127.31.12.126:44443)
0924 15:19:21.967672 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:21.967674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:21.967914 (+ 240us) server_negotiation.cc:407] Connection header received
0924 15:19:21.968390 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:21.968391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:21.968397 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:21.968454 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:21.969662 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:21.969663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:21.969663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:19:21.969667 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:21.969683 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:19:21.969687 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:21.969738 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:21.969831 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 15:19:21.969846 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 15:19:21.970106 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:21.967604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:44443 (local address 127.31.12.65:50427)
0924 15:19:21.967804 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:21.967834 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:21.968288 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:21.968522 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:21.968526 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:21.968541 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:21.969518 (+ 977us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:19:21.969519 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:21.969545 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:19:21.969570 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:19:21.969748 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:19:21.969753 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:19:21.969756 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:19:21.969840 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 15:19:21.969897 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":96,"threads_started":1}
I20260924 15:19:21.970589 9405 heartbeater.cc:347] Connected to a master server at 127.31.12.126:44443
I20260924 15:19:21.970664 9405 heartbeater.cc:464] Registering TS with master...
I20260924 15:19:21.970826 9405 heartbeater.cc:511] Master 127.31.12.126:44443 requested a full tablet report, sending...
I20260924 15:19:21.971256 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 9277
I20260924 15:19:21.971326 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:19:21.971382 9216 ts_manager.cc:194] Registered new tserver with Master: 9c8a1b0a76e74047b441bb380f2e58e2 (127.31.12.65:43433)
I20260924 15:19:21.971530 31793 external_mini_cluster.cc:1428] Setting key 98ee84a78002309c860405a95072e151
I20260924 15:19:21.972407 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:44443
--builtin_ntp_servers=127.31.12.84:42135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 15:19:22.052680 9408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:22.052920 9408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:22.052958 9408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:22.054174 9408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:22.054243 9408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:22.054287 9408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:19:22.055789 9408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:42135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:44443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:22.056031 9408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:22.056355 9408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:22.061290 9413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:22.061384 9408 server_base.cc:1034] running on GCE node
W20260924 15:19:22.061304 9414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:22.061306 9416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:22.061738 9408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:22.062004 9408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:22.063158 9408 hybrid_clock.cc:648] HybridClock initialized: now 1790263162063146 us; error 26 us; skew 500 ppm
I20260924 15:19:22.064333 9408 webserver.cc:460] Webserver started at http://127.31.12.66:36811/ using document root <none> and password file <none>
I20260924 15:19:22.064584 9408 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:22.064651 9408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:22.064805 9408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:19:22.065928 9408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "c2da758e93154364a4fa9a705b94a878"
format_stamp: "Formatted at 2026-09-24 15:19:22 on dist-test-slave-zwbb"
server_key: "0d323c8b2e1139d4fa7adc991a269584"
server_key_iv: "24f217c2410d7a4dd008b78606a25e4e"
server_key_version: "encryptionkey@0"
I20260924 15:19:22.066299 9408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "c2da758e93154364a4fa9a705b94a878"
format_stamp: "Formatted at 2026-09-24 15:19:22 on dist-test-slave-zwbb"
server_key: "0d323c8b2e1139d4fa7adc991a269584"
server_key_iv: "24f217c2410d7a4dd008b78606a25e4e"
server_key_version: "encryptionkey@0"
I20260924 15:19:22.067592 9408 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:19:22.068300 9422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:22.068475 9408 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:19:22.068579 9408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "c2da758e93154364a4fa9a705b94a878"
format_stamp: "Formatted at 2026-09-24 15:19:22 on dist-test-slave-zwbb"
server_key: "0d323c8b2e1139d4fa7adc991a269584"
server_key_iv: "24f217c2410d7a4dd008b78606a25e4e"
server_key_version: "encryptionkey@0"
I20260924 15:19:22.068655 9408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:22.072019 9408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:22.072284 9408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:22.072429 9408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:22.072641 9408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:19:22.072940 9408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:19:22.073001 9408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:22.073041 9408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:19:22.073068 9408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:22.078907 9408 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:35451
I20260924 15:19:22.078953 9535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:35451 every 8 connection(s)
I20260924 15:19:22.079248 9408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:19:22.081980 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:22.079959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:39839 (local address 127.31.12.126:44443)
0924 15:19:22.080004 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:22.080010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:22.080209 (+ 199us) server_negotiation.cc:407] Connection header received
0924 15:19:22.080620 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:22.080621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:22.080625 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:22.080689 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:22.081782 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:22.081784 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:22.081784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:19:22.081788 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:22.081803 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:19:22.081808 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:22.081865 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:22.081913 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 15:19:22.081925 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:19:22.082202 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:22.079904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:44443 (local address 127.31.12.66:39839)
0924 15:19:22.080085 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:22.080125 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:22.080546 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:22.080740 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:22.080743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:22.080750 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:22.081644 (+ 894us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:19:22.081645 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:22.081671 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:19:22.081697 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:19:22.081866 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:19:22.081869 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:19:22.081871 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:19:22.081921 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 15:19:22.081980 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260924 15:19:22.082705 9536 heartbeater.cc:347] Connected to a master server at 127.31.12.126:44443
I20260924 15:19:22.082795 9536 heartbeater.cc:464] Registering TS with master...
I20260924 15:19:22.082911 9536 heartbeater.cc:511] Master 127.31.12.126:44443 requested a full tablet report, sending...
I20260924 15:19:22.083256 9216 ts_manager.cc:194] Registered new tserver with Master: c2da758e93154364a4fa9a705b94a878 (127.31.12.66:35451)
I20260924 15:19:22.087096 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 9408
I20260924 15:19:22.087203 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:19:22.087410 31793 external_mini_cluster.cc:1428] Setting key 271816a1043b13fed050f6b3300cbfae
I20260924 15:19:22.088373 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:44443
--builtin_ntp_servers=127.31.12.84:42135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 15:19:22.169740 9539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:22.169940 9539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:22.169962 9539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:22.171155 9539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:22.171203 9539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:22.171231 9539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:19:22.172616 9539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:42135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:44443
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:22.172911 9539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:22.173209 9539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:22.178169 9547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:22.178176 9544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:22.178225 9539 server_base.cc:1034] running on GCE node
W20260924 15:19:22.178187 9545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:22.178587 9539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:22.178833 9539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:22.180013 9539 hybrid_clock.cc:648] HybridClock initialized: now 1790263162179983 us; error 34 us; skew 500 ppm
I20260924 15:19:22.181382 9539 webserver.cc:460] Webserver started at http://127.31.12.67:39769/ using document root <none> and password file <none>
I20260924 15:19:22.181639 9539 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:22.181712 9539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:22.181869 9539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:19:22.182935 9539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "096b5b46e3dc4fb5b3da30173a8d401a"
format_stamp: "Formatted at 2026-09-24 15:19:22 on dist-test-slave-zwbb"
server_key: "937b5862d3a2a44c5150f591bc003a19"
server_key_iv: "727fac30bf295714831464ac810af523"
server_key_version: "encryptionkey@0"
I20260924 15:19:22.183318 9539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "096b5b46e3dc4fb5b3da30173a8d401a"
format_stamp: "Formatted at 2026-09-24 15:19:22 on dist-test-slave-zwbb"
server_key: "937b5862d3a2a44c5150f591bc003a19"
server_key_iv: "727fac30bf295714831464ac810af523"
server_key_version: "encryptionkey@0"
I20260924 15:19:22.184651 9539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:19:22.185530 9553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:22.185727 9539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:19:22.185842 9539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "096b5b46e3dc4fb5b3da30173a8d401a"
format_stamp: "Formatted at 2026-09-24 15:19:22 on dist-test-slave-zwbb"
server_key: "937b5862d3a2a44c5150f591bc003a19"
server_key_iv: "727fac30bf295714831464ac810af523"
server_key_version: "encryptionkey@0"
I20260924 15:19:22.185956 9539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:22.189239 9539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:22.189559 9539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:22.189698 9539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:22.189899 9539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:19:22.190217 9539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:19:22.190277 9539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:22.190316 9539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:19:22.190342 9539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:22.195839 9539 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:34469
I20260924 15:19:22.195914 9668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:34469 every 8 connection(s)
I20260924 15:19:22.196241 9539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:19:22.199383 9255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:22.196989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:41389 (local address 127.31.12.126:44443)
0924 15:19:22.197048 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:22.197050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:22.197293 (+ 243us) server_negotiation.cc:407] Connection header received
0924 15:19:22.197853 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:22.197855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:22.197861 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:22.197908 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:22.199105 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:22.199106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:22.199106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:19:22.199111 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:22.199127 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:19:22.199131 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:22.199205 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:22.199297 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 15:19:22.199312 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:19:22.199596 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:22.196974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:44443 (local address 127.31.12.67:41389)
0924 15:19:22.197177 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:22.197208 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:22.197738 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:22.197996 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:22.197999 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:22.198006 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:22.198944 (+ 938us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:19:22.198944 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:22.198971 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:19:22.198997 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:19:22.199206 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:19:22.199212 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:19:22.199215 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:19:22.199307 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 15:19:22.199370 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":100,"threads_started":1}
I20260924 15:19:22.200078 9669 heartbeater.cc:347] Connected to a master server at 127.31.12.126:44443
I20260924 15:19:22.200168 9669 heartbeater.cc:464] Registering TS with master...
I20260924 15:19:22.200301 9669 heartbeater.cc:511] Master 127.31.12.126:44443 requested a full tablet report, sending...
I20260924 15:19:22.200662 9216 ts_manager.cc:194] Registered new tserver with Master: 096b5b46e3dc4fb5b3da30173a8d401a (127.31.12.67:34469)
I20260924 15:19:22.203203 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 9539
I20260924 15:19:22.203281 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:19:22.203436 31793 external_mini_cluster.cc:1428] Setting key b9517248f9888e667b7adfbb962a1033
I20260924 15:19:22.204510 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 15:19:22.206259 9255 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:19:22.205870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43802 (local address 127.31.12.126:44443)
0924 15:19:22.205904 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:22.205905 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:22.206038 (+ 133us) server_negotiation.cc:407] Connection header received
0924 15:19:22.206076 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:22.206077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:22.206081 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:22.206128 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:22.206217 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43802: BlockingRecv error: recv got EOF from 127.0.0.1:43802 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 15:19:22.206328 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 9277
I20260924 15:19:22.213284 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 9408
I20260924 15:19:22.219842 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 9539
I20260924 15:19:22.223953 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 9185
2026-09-24T15:19:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (562 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:19:22 dist-test-slave-zwbb krb5kdc[9679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:19:22 dist-test-slave-zwbb krb5kdc[9679](info): set up 2 sockets
Sep 24 15:19:22 dist-test-slave-zwbb krb5kdc[9679](info): commencing operation
krb5kdc: starting...
W20260924 15:19:23.275666 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:19:23 dist-test-slave-zwbb krb5kdc[9679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:19:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:19:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:19:23.393075 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:44919
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:44919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:19:23.475124 9692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:23.475374 9692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:23.475414 9692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:23.476581 9692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:19:23.476652 9692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:23.476677 9692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:23.476699 9692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:19:23.476730 9692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:19:23.478331 9692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:44919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:44919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:23.478610 9692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:23.478855 9692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:23.483933 9700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:23.483940 9698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:23.483942 9697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:23.483963 9692 server_base.cc:1034] running on GCE node
I20260924 15:19:23.484457 9692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:23.484807 9692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:23.485989 9692 hybrid_clock.cc:648] HybridClock initialized: now 1790263163485975 us; error 30 us; skew 500 ppm
Sep 24 15:19:23 dist-test-slave-zwbb krb5kdc[9679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263163, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:19:23.488572 9692 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:19:23.488950 9692 webserver.cc:460] Webserver started at http://127.31.12.126:39117/ using document root <none> and password file <none>
I20260924 15:19:23.489149 9692 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:23.489190 9692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:23.489279 9692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:19:23.490350 9692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "7478903466a34dd2bba7493117cb998e"
format_stamp: "Formatted at 2026-09-24 15:19:23 on dist-test-slave-zwbb"
server_key: "ca15cd4c30ad4befa82394594be26e39"
server_key_iv: "fe7a361f0b4479e53ac5780e4926d478"
server_key_version: "encryptionkey@0"
I20260924 15:19:23.490670 9692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "7478903466a34dd2bba7493117cb998e"
format_stamp: "Formatted at 2026-09-24 15:19:23 on dist-test-slave-zwbb"
server_key: "ca15cd4c30ad4befa82394594be26e39"
server_key_iv: "fe7a361f0b4479e53ac5780e4926d478"
server_key_version: "encryptionkey@0"
I20260924 15:19:23.491905 9692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:19:23.492589 9707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:23.492807 9692 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:19:23.492916 9692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "7478903466a34dd2bba7493117cb998e"
format_stamp: "Formatted at 2026-09-24 15:19:23 on dist-test-slave-zwbb"
server_key: "ca15cd4c30ad4befa82394594be26e39"
server_key_iv: "fe7a361f0b4479e53ac5780e4926d478"
server_key_version: "encryptionkey@0"
I20260924 15:19:23.492998 9692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:23.503436 9692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:23.505366 9692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:23.505566 9692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:23.509258 9692 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:44919
I20260924 15:19:23.509307 9760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:44919 every 8 connection(s)
I20260924 15:19:23.509707 9692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:19:23.510710 9761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:23.512984 9761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e: Bootstrap starting.
I20260924 15:19:23.513638 9761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:23.513932 9761 log.cc:824] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:19:23.514631 9761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e: No bootstrap required, opened a new log
I20260924 15:19:23.516242 9761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7478903466a34dd2bba7493117cb998e" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44919 } }
I20260924 15:19:23.516404 9761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:23.516456 9761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7478903466a34dd2bba7493117cb998e, State: Initialized, Role: FOLLOWER
I20260924 15:19:23.516618 9761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7478903466a34dd2bba7493117cb998e" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44919 } }
I20260924 15:19:23.516716 9761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:19:23.516760 9761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:19:23.516809 9761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:23.517706 9761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7478903466a34dd2bba7493117cb998e" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44919 } }
I20260924 15:19:23.517838 9761 leader_election.cc:302] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7478903466a34dd2bba7493117cb998e; no voters:
I20260924 15:19:23.517937 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 9692
I20260924 15:19:23.518033 9761 leader_election.cc:288] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:19:23.518042 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:19:23.518080 9764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:19:23.518239 9764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e [term 1 LEADER]: Becoming Leader. State: Replica: 7478903466a34dd2bba7493117cb998e, State: Running, Role: LEADER
I20260924 15:19:23.518247 31793 external_mini_cluster.cc:1428] Setting key e03fe7661a8761c58209be7361c84413
I20260924 15:19:23.518291 9761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:19:23.518370 9764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7478903466a34dd2bba7493117cb998e" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44919 } }
I20260924 15:19:23.518858 9765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7478903466a34dd2bba7493117cb998e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7478903466a34dd2bba7493117cb998e" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44919 } } }
I20260924 15:19:23.518944 9765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e [sys.catalog]: This master's current role is: LEADER
I20260924 15:19:23.519142 9766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7478903466a34dd2bba7493117cb998e. Latest consensus state: current_term: 1 leader_uuid: "7478903466a34dd2bba7493117cb998e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7478903466a34dd2bba7493117cb998e" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44919 } } }
I20260924 15:19:23.519209 9766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e [sys.catalog]: This master's current role is: LEADER
I20260924 15:19:23.519433 9775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:19:23.520004 9775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:19:23.522039 9775 catalog_manager.cc:1347] Generated new cluster ID: c5930edffe33498bb1c6a92d6fb72f96
I20260924 15:19:23.522096 9775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:19:23 dist-test-slave-zwbb krb5kdc[9679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:19:23.530592 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:23.518836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36560 (local address 127.31.12.126:44919)
0924 15:19:23.519006 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:23.519016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:23.519052 (+ 36us) server_negotiation.cc:407] Connection header received
0924 15:19:23.519186 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:23.519193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:23.519247 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:23.519401 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:23.520195 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:23.521769 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:23.522817 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:23.523500 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:23.523616 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:23.526411 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:23.526415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:23.526416 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:19:23.526428 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:23.528466 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:23.528836 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:23.528839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:23.528839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:23.528972 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:23.529161 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:23.529161 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:23.529162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:23.529499 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:23.530072 (+ 573us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:23.530156 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 15:19:23.530313 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
I20260924 15:19:23.548877 9775 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:19:23.549517 9775 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:19:23.559798 9775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e: Generated new TSK 0
I20260924 15:19:23.560071 9775 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
I20260924 15:19:23.592607 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:44919
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:19:23.674499 9787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:23.674716 9787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:23.674741 9787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:23.675928 9787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:23.675974 9787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:23.676003 9787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:19:23.677505 9787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:44919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:23.677757 9787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:23.677997 9787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:23.683004 9793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:23.682972 9796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:23.683027 9787 server_base.cc:1034] running on GCE node
W20260924 15:19:23.682965 9794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:23.683372 9787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:23.683633 9787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:23.684813 9787 hybrid_clock.cc:648] HybridClock initialized: now 1790263163684813 us; error 40 us; skew 500 ppm
Sep 24 15:19:23 dist-test-slave-zwbb krb5kdc[9679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263163, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:19:23.687304 9787 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:19:23.687804 9787 webserver.cc:460] Webserver started at http://127.31.12.65:35535/ using document root <none> and password file <none>
I20260924 15:19:23.688035 9787 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:23.688105 9787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:23.688258 9787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:19:23.689399 9787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "ee73ffbe183f4d5eb44bea3e70da6a68"
format_stamp: "Formatted at 2026-09-24 15:19:23 on dist-test-slave-zwbb"
server_key: "9e6a4b512f98c98c62ec31c6f2347bd9"
server_key_iv: "7099350a8687836d1781f2bd973c81cc"
server_key_version: "encryptionkey@0"
I20260924 15:19:23.689832 9787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "ee73ffbe183f4d5eb44bea3e70da6a68"
format_stamp: "Formatted at 2026-09-24 15:19:23 on dist-test-slave-zwbb"
server_key: "9e6a4b512f98c98c62ec31c6f2347bd9"
server_key_iv: "7099350a8687836d1781f2bd973c81cc"
server_key_version: "encryptionkey@0"
I20260924 15:19:23.691255 9787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:19:23.692176 9803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:23.692399 9787 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:19:23.692520 9787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "ee73ffbe183f4d5eb44bea3e70da6a68"
format_stamp: "Formatted at 2026-09-24 15:19:23 on dist-test-slave-zwbb"
server_key: "9e6a4b512f98c98c62ec31c6f2347bd9"
server_key_iv: "7099350a8687836d1781f2bd973c81cc"
server_key_version: "encryptionkey@0"
I20260924 15:19:23.692612 9787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:23.711799 9787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:23.713904 9787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:23.714069 9787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:23.714318 9787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:19:23.714653 9787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:19:23.714707 9787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:23.714748 9787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:19:23.714778 9787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:23.721511 9787 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:33687
I20260924 15:19:23.721556 9918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:33687 every 8 connection(s)
I20260924 15:19:23.721889 9787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:19:23 dist-test-slave-zwbb krb5kdc[9679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263163, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:19:23.728276 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 9787
I20260924 15:19:23.728387 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:19:23.728629 31793 external_mini_cluster.cc:1428] Setting key b440617b05b2e3a648c61becd81e51f3
I20260924 15:19:23.731930 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:23.722601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:44551 (local address 127.31.12.126:44919)
0924 15:19:23.722665 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:23.722666 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:23.722891 (+ 225us) server_negotiation.cc:407] Connection header received
0924 15:19:23.723388 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:23.723389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:23.723403 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:23.723472 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:23.724404 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:23.724930 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:23.726005 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:23.726205 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:23.726262 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:23.728711 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:23.728713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:23.728714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:19:23.728722 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:23.730299 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:23.730849 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:23.730850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:23.730851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:23.730922 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:23.731225 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:23.731226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:23.731227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:23.731376 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:23.731458 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:23.731709 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 15:19:23.731801 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 15:19:23.731976 9921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:23.722573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:44919 (local address 127.31.12.65:44551)
0924 15:19:23.722751 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:23.722793 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:23.723307 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:23.723536 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:23.723539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:23.724129 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:23.724325 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:23.724330 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:23.725016 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:23.725016 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:23.725922 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:23.725924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:23.726282 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:23.726283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:23.726382 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:23.726887 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:23.726895 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:23.728592 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:19:23.730384 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:23.730392 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:23.730394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:23.730734 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:23.731002 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:23.731003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:23.731004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:23.731156 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:23.731455 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:19:23.731460 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:19:23.731578 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 15:19:23.731663 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 15:19:23.731784 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260924 15:19:23.732538 9919 heartbeater.cc:347] Connected to a master server at 127.31.12.126:44919
I20260924 15:19:23.732641 9919 heartbeater.cc:464] Registering TS with master...
I20260924 15:19:23.732846 9919 heartbeater.cc:511] Master 127.31.12.126:44919 requested a full tablet report, sending...
I20260924 15:19:23.733376 9725 ts_manager.cc:194] Registered new tserver with Master: ee73ffbe183f4d5eb44bea3e70da6a68 (127.31.12.65:33687)
I20260924 15:19:23.734077 9725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:44551
WARNING: no policy specified for kudu/127.31.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
I20260924 15:19:23.754302 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:44919
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:19:23.835549 9924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:23.835752 9924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:23.835775 9924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:23.836911 9924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:23.836951 9924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:23.836977 9924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:19:23.838459 9924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:44919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:23.838707 9924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:23.838956 9924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:23.843971 9929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:23.843921 9930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:23.843921 9932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:23.844197 9924 server_base.cc:1034] running on GCE node
I20260924 15:19:23.844419 9924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:23.844700 9924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:23.845863 9924 hybrid_clock.cc:648] HybridClock initialized: now 1790263163845857 us; error 34 us; skew 500 ppm
Sep 24 15:19:23 dist-test-slave-zwbb krb5kdc[9679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263163, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:19:23.848374 9924 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:19:23.848814 9924 webserver.cc:460] Webserver started at http://127.31.12.66:45317/ using document root <none> and password file <none>
I20260924 15:19:23.849004 9924 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:23.849044 9924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:23.849130 9924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:19:23.850168 9924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "8125f364d96741b88c77034821be151c"
format_stamp: "Formatted at 2026-09-24 15:19:23 on dist-test-slave-zwbb"
server_key: "7962a0587ea48a29e198e9bb919de960"
server_key_iv: "ed02eabb8997c0f9a4268dc093149597"
server_key_version: "encryptionkey@0"
I20260924 15:19:23.850463 9924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "8125f364d96741b88c77034821be151c"
format_stamp: "Formatted at 2026-09-24 15:19:23 on dist-test-slave-zwbb"
server_key: "7962a0587ea48a29e198e9bb919de960"
server_key_iv: "ed02eabb8997c0f9a4268dc093149597"
server_key_version: "encryptionkey@0"
I20260924 15:19:23.851675 9924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:19:23.852349 9939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:23.852540 9924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:19:23.852631 9924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "8125f364d96741b88c77034821be151c"
format_stamp: "Formatted at 2026-09-24 15:19:23 on dist-test-slave-zwbb"
server_key: "7962a0587ea48a29e198e9bb919de960"
server_key_iv: "ed02eabb8997c0f9a4268dc093149597"
server_key_version: "encryptionkey@0"
I20260924 15:19:23.852705 9924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:23.868643 9924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:23.870899 9924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:23.871047 9924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:23.871281 9924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:19:23.871585 9924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:19:23.871639 9924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:23.871670 9924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:19:23.871697 9924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:23.877303 9924 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:35585
I20260924 15:19:23.877350 10053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:35585 every 8 connection(s)
I20260924 15:19:23.877681 9924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:19:23.879323 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 9924
I20260924 15:19:23.879417 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:19:23.879612 31793 external_mini_cluster.cc:1428] Setting key 53488a72548ea003cbb2c391bbb7c34a
Sep 24 15:19:23 dist-test-slave-zwbb krb5kdc[9679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263163, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.12.67@KRBTEST.COM; defaulting to no policy
I20260924 15:19:23.887485 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:23.878450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:43981 (local address 127.31.12.126:44919)
0924 15:19:23.878509 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:23.878511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:23.878724 (+ 213us) server_negotiation.cc:407] Connection header received
0924 15:19:23.879207 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:23.879208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:23.879222 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:23.879312 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:23.880571 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:23.881331 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:23.882390 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:23.882604 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:23.882690 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:23.884722 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:23.884723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:23.884723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:19:23.884731 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:23.886136 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:23.886544 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:23.886545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:23.886546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:23.886606 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:23.886857 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:23.886858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:23.886858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:23.886978 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:23.887054 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:23.887313 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 15:19:23.887393 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 15:19:23.887674 10056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:23.878415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:44919 (local address 127.31.12.66:43981)
0924 15:19:23.878588 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:23.878643 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:23.879104 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:23.879419 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:23.879424 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:23.880195 (+ 771us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:23.880458 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:23.880466 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:23.881433 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:23.881434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:23.882283 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:23.882285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:23.882699 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:23.882700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:23.882833 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:23.883318 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:23.883326 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:23.884614 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:19:23.886237 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:23.886245 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:23.886247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:23.886463 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:23.886673 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:23.886674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:23.886674 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:23.886781 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:23.887045 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:19:23.887052 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:19:23.887186 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 15:19:23.887293 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 15:19:23.887442 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":86,"threads_started":1}
I20260924 15:19:23.888126 10054 heartbeater.cc:347] Connected to a master server at 127.31.12.126:44919
I20260924 15:19:23.888208 10054 heartbeater.cc:464] Registering TS with master...
I20260924 15:19:23.888392 10054 heartbeater.cc:511] Master 127.31.12.126:44919 requested a full tablet report, sending...
I20260924 15:19:23.888738 9725 ts_manager.cc:194] Registered new tserver with Master: 8125f364d96741b88c77034821be151c (127.31.12.66:35585)
I20260924 15:19:23.889153 9725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:43981
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
I20260924 15:19:23.904362 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:44919
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:19:23.985244 10059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:19:23.985495 10059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:19:23.985538 10059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:19:23.986645 10059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:19:23.986682 10059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:19:23.986708 10059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:19:23.988085 10059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:44919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:19:23.988328 10059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:19:23.988569 10059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:19:23.993582 10065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:23.993577 10064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:19:23.993578 10067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:19:23.993739 10059 server_base.cc:1034] running on GCE node
I20260924 15:19:23.993949 10059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:19:23.994182 10059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:19:23.995319 10059 hybrid_clock.cc:648] HybridClock initialized: now 1790263163995308 us; error 26 us; skew 500 ppm
Sep 24 15:19:23 dist-test-slave-zwbb krb5kdc[9679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263163, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:19:23.997910 10059 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:19:23.998363 10059 webserver.cc:460] Webserver started at http://127.31.12.67:39647/ using document root <none> and password file <none>
I20260924 15:19:23.998607 10059 fs_manager.cc:359] Metadata directory not provided
I20260924 15:19:23.998675 10059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:19:23.998821 10059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:19:23.999841 10059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "283029ffbace4ce1aa3be8387e25a800"
format_stamp: "Formatted at 2026-09-24 15:19:23 on dist-test-slave-zwbb"
server_key: "d42568b306a5c972937490bb3f2df605"
server_key_iv: "5bb1cb183661d67b01df9f3d3f0f143c"
server_key_version: "encryptionkey@0"
I20260924 15:19:24.000201 10059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "283029ffbace4ce1aa3be8387e25a800"
format_stamp: "Formatted at 2026-09-24 15:19:23 on dist-test-slave-zwbb"
server_key: "d42568b306a5c972937490bb3f2df605"
server_key_iv: "5bb1cb183661d67b01df9f3d3f0f143c"
server_key_version: "encryptionkey@0"
I20260924 15:19:24.001499 10059 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:19:24.002218 10074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:24.002377 10059 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:19:24.002478 10059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "283029ffbace4ce1aa3be8387e25a800"
format_stamp: "Formatted at 2026-09-24 15:19:23 on dist-test-slave-zwbb"
server_key: "d42568b306a5c972937490bb3f2df605"
server_key_iv: "5bb1cb183661d67b01df9f3d3f0f143c"
server_key_version: "encryptionkey@0"
I20260924 15:19:24.002560 10059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:19:24.019213 10059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:19:24.021136 10059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:19:24.021306 10059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:19:24.021595 10059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:19:24.021903 10059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:19:24.021956 10059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:24.021991 10059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:19:24.022022 10059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:19:24.027972 10059 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:43271
I20260924 15:19:24.028023 10190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:43271 every 8 connection(s)
I20260924 15:19:24.028321 10059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:19:24.028801 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 10059
I20260924 15:19:24.028913 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:19:24.029148 31793 external_mini_cluster.cc:1428] Setting key fe0f42992c8fe358b95eba911507dc2f
Sep 24 15:19:24 dist-test-slave-zwbb krb5kdc[9679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263163, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:19:24.037177 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:24.028967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:41247 (local address 127.31.12.126:44919)
0924 15:19:24.029017 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:24.029019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:24.029304 (+ 285us) server_negotiation.cc:407] Connection header received
0924 15:19:24.030221 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:24.030222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:24.030232 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:24.030285 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:24.031192 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.031668 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.032596 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.032771 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.032820 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:24.034897 (+ 2077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:24.034898 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:24.034898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:19:24.034905 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:24.036074 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:24.036414 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:24.036414 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:24.036415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:24.036469 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:24.036669 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:24.036669 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:24.036669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:24.036768 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:24.036817 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:24.037051 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 15:19:24.037114 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:19:24.037371 10193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:24.028986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:44919 (local address 127.31.12.67:41247)
0924 15:19:24.029152 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:24.029204 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:24.029916 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:24.030363 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:24.030366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:24.030922 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:19:24.031114 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:24.031119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.031742 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.031743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:24.032524 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:24.032526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.032839 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.032840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:24.032933 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:24.033484 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:19:24.033494 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:19:24.034816 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:19:24.036144 (+ 1328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:24.036149 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:24.036151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:24.036352 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:24.036525 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:19:24.036525 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:19:24.036526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:19:24.036617 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:19:24.036827 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:19:24.036831 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:19:24.036942 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 15:19:24.037023 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 15:19:24.037133 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":87,"threads_started":1}
I20260924 15:19:24.037807 10191 heartbeater.cc:347] Connected to a master server at 127.31.12.126:44919
I20260924 15:19:24.037910 10191 heartbeater.cc:464] Registering TS with master...
I20260924 15:19:24.038079 10191 heartbeater.cc:511] Master 127.31.12.126:44919 requested a full tablet report, sending...
I20260924 15:19:24.038444 9725 ts_manager.cc:194] Registered new tserver with Master: 283029ffbace4ce1aa3be8387e25a800 (127.31.12.67:43271)
I20260924 15:19:24.038774 9725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:41247
I20260924 15:19:24.040745 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:19:24.047264 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:24.041978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36572 (local address 127.31.12.126:44919)
0924 15:19:24.042038 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:24.042040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:24.042153 (+ 113us) server_negotiation.cc:407] Connection header received
0924 15:19:24.042175 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:24.042176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:24.042183 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:24.042233 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:19:24.042719 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.043167 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.043828 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.044007 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.044054 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:24.045149 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:19:24.045150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:19:24.045151 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:19:24.045155 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:19:24.046334 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:24.046633 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:24.046633 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:24.046634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:24.046681 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:19:24.046836 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:19:24.046836 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:19:24.046836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:19:24.046973 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:19:24.047043 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:24.047125 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 15:19:24.047180 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 15:19:24.048851 9725 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:19:24.050542 9725 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:19:24.057211 10200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:24.052585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:35585 (local address 127.0.0.1:58634)
0924 15:19:24.052828 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:24.052882 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:24.052935 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:24.053589 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:24.053591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:24.053593 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:19:24.053632 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:24.053633 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.054936 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.054937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:24.055999 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:24.056002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.056875 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.056876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:24.056959 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:24.056981 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 15:19:24.057069 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 15:19:24.057125 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":61,"thread_start_us":51,"threads_started":1}
I20260924 15:19:24.057214 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:24.052836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58634 (local address 127.31.12.66:35585)
0924 15:19:24.053394 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:24.053399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:24.053410 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:19:24.053472 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:24.053474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:24.053491 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:24.053587 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:19:24.053735 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.054829 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.056089 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.056775 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.056849 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:24.056905 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:24.057094 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 15:19:24.057134 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":90,"threads_started":1}
I20260924 15:19:24.057670 10201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:24.052592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:33687 (local address 127.0.0.1:52256)
0924 15:19:24.052917 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:24.052928 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:24.052983 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:24.054009 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:24.054011 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:24.054013 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:19:24.054076 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:24.054078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.055040 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.055040 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:24.056353 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:24.056355 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.057377 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.057378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:24.057484 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:24.057489 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:19:24.057546 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:19:24.057591 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":6,"thread_start_us":47,"threads_started":1}
I20260924 15:19:24.057701 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:24.053088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.31.12.65:33687)
0924 15:19:24.053751 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:24.053756 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:24.053767 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:19:24.053807 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:24.053810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:24.053826 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:24.053905 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:19:24.054161 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.054930 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.056461 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.057214 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.057283 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:24.057341 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:24.057566 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 15:19:24.057621 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":117,"threads_started":1}
I20260924 15:19:24.057796 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:24.052559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:43271 (local address 127.0.0.1:42490)
0924 15:19:24.052827 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:24.052867 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:24.052935 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:24.053378 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:24.053381 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:24.053388 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:19:24.053553 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:24.053556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.055029 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.055030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:24.056352 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:24.056355 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.057189 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.057189 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:24.057249 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:24.057252 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:19:24.057630 (+ 378us) client_negotiation.cc:240] Negotiation successful
0924 15:19:24.057675 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":86,"thread_start_us":65,"threads_started":1}
I20260924 15:19:24.058012 9988 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac1cc9b7102438bb406c3ab920b4f35 (DEFAULT_TABLE table=test-table [id=96b1579e58fd47f190f1aeff765fea0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:19:24.058111 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:24.052633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42490 (local address 127.31.12.67:43271)
0924 15:19:24.053069 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:24.053076 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:24.053105 (+ 29us) server_negotiation.cc:407] Connection header received
0924 15:19:24.053144 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:24.053146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:24.053162 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:24.053238 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:19:24.053657 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.054932 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.056451 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.057039 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.057079 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:24.057131 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:24.058026 (+ 895us) server_negotiation.cc:299] Negotiation successful
0924 15:19:24.058060 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":83,"threads_started":1}
I20260924 15:19:24.058403 9988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac1cc9b7102438bb406c3ab920b4f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:24.060950 10205 tablet_bootstrap.cc:492] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c: Bootstrap starting.
I20260924 15:19:24.061655 10205 tablet_bootstrap.cc:654] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:24.061668 9851 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac1cc9b7102438bb406c3ab920b4f35 (DEFAULT_TABLE table=test-table [id=96b1579e58fd47f190f1aeff765fea0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:19:24.061815 10123 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac1cc9b7102438bb406c3ab920b4f35 (DEFAULT_TABLE table=test-table [id=96b1579e58fd47f190f1aeff765fea0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:19:24.062021 9851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac1cc9b7102438bb406c3ab920b4f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:24.062134 10123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac1cc9b7102438bb406c3ab920b4f35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:19:24.062179 10205 log.cc:824] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c: Log is configured to *not* fsync() on all Append() calls
I20260924 15:19:24.062929 10205 tablet_bootstrap.cc:492] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c: No bootstrap required, opened a new log
I20260924 15:19:24.063000 10205 ts_tablet_manager.cc:1397] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:19:24.064870 10207 tablet_bootstrap.cc:492] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800: Bootstrap starting.
I20260924 15:19:24.065039 10205 raft_consensus.cc:348] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283029ffbace4ce1aa3be8387e25a800" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 43271 } } peers { permanent_uuid: "8125f364d96741b88c77034821be151c" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35585 } } peers { permanent_uuid: "ee73ffbe183f4d5eb44bea3e70da6a68" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 33687 } }
I20260924 15:19:24.065218 10205 raft_consensus.cc:374] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:24.065258 10205 raft_consensus.cc:729] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8125f364d96741b88c77034821be151c, State: Initialized, Role: FOLLOWER
I20260924 15:19:24.065373 10205 consensus_queue.cc:260] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283029ffbace4ce1aa3be8387e25a800" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 43271 } } peers { permanent_uuid: "8125f364d96741b88c77034821be151c" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35585 } } peers { permanent_uuid: "ee73ffbe183f4d5eb44bea3e70da6a68" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 33687 } }
I20260924 15:19:24.065479 10208 tablet_bootstrap.cc:492] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68: Bootstrap starting.
I20260924 15:19:24.065582 10207 tablet_bootstrap.cc:654] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:24.065697 10205 ts_tablet_manager.cc:1428] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:19:24.065842 10054 heartbeater.cc:503] Master 127.31.12.126:44919 was elected leader, sending a full tablet report...
I20260924 15:19:24.066143 10207 log.cc:824] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800: Log is configured to *not* fsync() on all Append() calls
I20260924 15:19:24.066236 10208 tablet_bootstrap.cc:654] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68: Neither blocks nor log segments found. Creating new log.
I20260924 15:19:24.066787 10208 log.cc:824] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68: Log is configured to *not* fsync() on all Append() calls
I20260924 15:19:24.066936 10207 tablet_bootstrap.cc:492] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800: No bootstrap required, opened a new log
I20260924 15:19:24.067010 10207 ts_tablet_manager.cc:1397] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:19:24.067512 10208 tablet_bootstrap.cc:492] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68: No bootstrap required, opened a new log
I20260924 15:19:24.067593 10208 ts_tablet_manager.cc:1397] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:19:24.068578 10207 raft_consensus.cc:348] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283029ffbace4ce1aa3be8387e25a800" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 43271 } } peers { permanent_uuid: "8125f364d96741b88c77034821be151c" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35585 } } peers { permanent_uuid: "ee73ffbe183f4d5eb44bea3e70da6a68" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 33687 } }
I20260924 15:19:24.068691 10207 raft_consensus.cc:374] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:24.068711 10207 raft_consensus.cc:729] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 283029ffbace4ce1aa3be8387e25a800, State: Initialized, Role: FOLLOWER
I20260924 15:19:24.068821 10207 consensus_queue.cc:260] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283029ffbace4ce1aa3be8387e25a800" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 43271 } } peers { permanent_uuid: "8125f364d96741b88c77034821be151c" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35585 } } peers { permanent_uuid: "ee73ffbe183f4d5eb44bea3e70da6a68" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 33687 } }
I20260924 15:19:24.069095 10207 ts_tablet_manager.cc:1428] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:19:24.069108 10191 heartbeater.cc:503] Master 127.31.12.126:44919 was elected leader, sending a full tablet report...
I20260924 15:19:24.069029 10208 raft_consensus.cc:348] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283029ffbace4ce1aa3be8387e25a800" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 43271 } } peers { permanent_uuid: "8125f364d96741b88c77034821be151c" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35585 } } peers { permanent_uuid: "ee73ffbe183f4d5eb44bea3e70da6a68" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 33687 } }
I20260924 15:19:24.069200 10208 raft_consensus.cc:374] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:19:24.069245 10208 raft_consensus.cc:729] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee73ffbe183f4d5eb44bea3e70da6a68, State: Initialized, Role: FOLLOWER
I20260924 15:19:24.069360 10208 consensus_queue.cc:260] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283029ffbace4ce1aa3be8387e25a800" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 43271 } } peers { permanent_uuid: "8125f364d96741b88c77034821be151c" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35585 } } peers { permanent_uuid: "ee73ffbe183f4d5eb44bea3e70da6a68" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 33687 } }
I20260924 15:19:24.069648 10208 ts_tablet_manager.cc:1428] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:19:24.069681 9919 heartbeater.cc:503] Master 127.31.12.126:44919 was elected leader, sending a full tablet report...
I20260924 15:19:24.087694 10213 raft_consensus.cc:482] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:19:24.087800 10213 raft_consensus.cc:504] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283029ffbace4ce1aa3be8387e25a800" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 43271 } } peers { permanent_uuid: "8125f364d96741b88c77034821be151c" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35585 } } peers { permanent_uuid: "ee73ffbe183f4d5eb44bea3e70da6a68" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 33687 } }
I20260924 15:19:24.088236 10213 leader_election.cc:288] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 283029ffbace4ce1aa3be8387e25a800 (127.31.12.67:43271), 8125f364d96741b88c77034821be151c (127.31.12.66:35585)
I20260924 15:19:24.091096 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:24.088487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:48513 (local address 127.31.12.67:43271)
0924 15:19:24.088548 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:24.088551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:24.088560 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:19:24.088588 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:24.088588 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:24.088598 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:24.088659 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:19:24.088824 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.089281 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.090370 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.090736 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.090833 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:24.090842 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:24.090994 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 15:19:24.091033 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:19:24.091113 9921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:24.088383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:43271 (local address 127.31.12.65:48513)
0924 15:19:24.088441 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:24.088453 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:24.088501 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:24.088665 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:24.088666 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:24.088668 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:19:24.088743 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:24.088745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.089371 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.089372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:24.090200 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:24.090202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.090833 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.090834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:24.090919 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:24.090922 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:19:24.090989 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 15:19:24.091031 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 15:19:24.091189 10203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:24.088553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:40835 (local address 127.31.12.66:35585)
0924 15:19:24.088598 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:24.088600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:24.088644 (+ 44us) server_negotiation.cc:407] Connection header received
0924 15:19:24.088703 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:24.088704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:24.088713 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:24.088790 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:19:24.088953 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.089403 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.090426 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.090805 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.090912 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:24.090919 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:24.091081 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 15:19:24.091122 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 15:19:24.091455 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:24.088426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:35585 (local address 127.31.12.65:40835)
0924 15:19:24.088598 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:24.088607 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:19:24.088654 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:19:24.088793 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:19:24.088794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:19:24.088795 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:19:24.088875 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:24.088876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.089534 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.089535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:24.090320 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:19:24.090323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.090918 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.090919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:19:24.090997 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:24.091001 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:19:24.091063 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 15:19:24.091100 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":105,"threads_started":1}
I20260924 15:19:24.091578 10144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac1cc9b7102438bb406c3ab920b4f35" candidate_uuid: "ee73ffbe183f4d5eb44bea3e70da6a68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283029ffbace4ce1aa3be8387e25a800" is_pre_election: true
I20260924 15:19:24.091800 10144 raft_consensus.cc:2454] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee73ffbe183f4d5eb44bea3e70da6a68 in term 0.
I20260924 15:19:24.091782 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac1cc9b7102438bb406c3ab920b4f35" candidate_uuid: "ee73ffbe183f4d5eb44bea3e70da6a68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125f364d96741b88c77034821be151c" is_pre_election: true
I20260924 15:19:24.091913 10008 raft_consensus.cc:2454] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee73ffbe183f4d5eb44bea3e70da6a68 in term 0.
I20260924 15:19:24.092015 9806 leader_election.cc:302] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 283029ffbace4ce1aa3be8387e25a800, ee73ffbe183f4d5eb44bea3e70da6a68; no voters:
I20260924 15:19:24.092197 10213 raft_consensus.cc:2781] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:19:24.092252 10213 raft_consensus.cc:482] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:19:24.092280 10213 raft_consensus.cc:3037] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:24.092949 10213 raft_consensus.cc:504] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283029ffbace4ce1aa3be8387e25a800" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 43271 } } peers { permanent_uuid: "8125f364d96741b88c77034821be151c" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35585 } } peers { permanent_uuid: "ee73ffbe183f4d5eb44bea3e70da6a68" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 33687 } }
I20260924 15:19:24.093088 10213 leader_election.cc:288] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68 [CANDIDATE]: Term 1 election: Requested vote from peers 283029ffbace4ce1aa3be8387e25a800 (127.31.12.67:43271), 8125f364d96741b88c77034821be151c (127.31.12.66:35585)
I20260924 15:19:24.093245 10008 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac1cc9b7102438bb406c3ab920b4f35" candidate_uuid: "ee73ffbe183f4d5eb44bea3e70da6a68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8125f364d96741b88c77034821be151c"
I20260924 15:19:24.093286 10144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ac1cc9b7102438bb406c3ab920b4f35" candidate_uuid: "ee73ffbe183f4d5eb44bea3e70da6a68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "283029ffbace4ce1aa3be8387e25a800"
I20260924 15:19:24.093318 10008 raft_consensus.cc:3037] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:24.093343 10144 raft_consensus.cc:3037] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:19:24.094238 10008 raft_consensus.cc:2454] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee73ffbe183f4d5eb44bea3e70da6a68 in term 1.
I20260924 15:19:24.094238 10144 raft_consensus.cc:2454] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee73ffbe183f4d5eb44bea3e70da6a68 in term 1.
I20260924 15:19:24.094478 9807 leader_election.cc:302] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8125f364d96741b88c77034821be151c, ee73ffbe183f4d5eb44bea3e70da6a68; no voters:
I20260924 15:19:24.094610 10213 raft_consensus.cc:2781] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:19:24.094749 10213 raft_consensus.cc:686] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68 [term 1 LEADER]: Becoming Leader. State: Replica: ee73ffbe183f4d5eb44bea3e70da6a68, State: Running, Role: LEADER
I20260924 15:19:24.094848 10213 consensus_queue.cc:237] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283029ffbace4ce1aa3be8387e25a800" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 43271 } } peers { permanent_uuid: "8125f364d96741b88c77034821be151c" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35585 } } peers { permanent_uuid: "ee73ffbe183f4d5eb44bea3e70da6a68" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 33687 } }
I20260924 15:19:24.095676 9725 catalog_manager.cc:5495] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee73ffbe183f4d5eb44bea3e70da6a68 (127.31.12.65). New cstate: current_term: 1 leader_uuid: "ee73ffbe183f4d5eb44bea3e70da6a68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "283029ffbace4ce1aa3be8387e25a800" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 43271 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8125f364d96741b88c77034821be151c" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee73ffbe183f4d5eb44bea3e70da6a68" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 33687 } health_report { overall_health: HEALTHY } } }
I20260924 15:19:24.104918 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:24.102679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52266 (local address 127.31.12.65:33687)
0924 15:19:24.102719 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:19:24.102721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:19:24.102728 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:19:24.102789 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:19:24.102790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:19:24.102805 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:19:24.102872 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:19:24.102987 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.103416 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.104225 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:19:24.104413 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:19:24.104466 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:19:24.104598 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:19:24.104682 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:19:24.104737 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 15:19:24.104773 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 15:19:24.104787 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:19:24.106669 10144 raft_consensus.cc:1261] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800 [term 1 FOLLOWER]: Refusing update from remote peer ee73ffbe183f4d5eb44bea3e70da6a68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:19:24.106683 10008 raft_consensus.cc:1261] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c [term 1 FOLLOWER]: Refusing update from remote peer ee73ffbe183f4d5eb44bea3e70da6a68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:19:24.107002 10213 consensus_queue.cc:1035] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "283029ffbace4ce1aa3be8387e25a800" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 43271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:19:24.107180 10215 consensus_queue.cc:1035] T 8ac1cc9b7102438bb406c3ab920b4f35 P ee73ffbe183f4d5eb44bea3e70da6a68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8125f364d96741b88c77034821be151c" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:19:24.109931 10219 mvcc.cc:205] Tried to move back new op lower bound from 7332917920178434048 to 7332917920133160960. Current Snapshot: MvccSnapshot[applied={T|T < 7332917920178434048}]
I20260924 15:19:24.111769 9724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:19:24.111853 9724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:19:24.112630 9724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7478903466a34dd2bba7493117cb998e: Sending DeleteTablet for 3 replicas of tablet 8ac1cc9b7102438bb406c3ab920b4f35
I20260924 15:19:24.112963 9851 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ac1cc9b7102438bb406c3ab920b4f35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:19:24 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:52256
I20260924 15:19:24.112988 9988 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ac1cc9b7102438bb406c3ab920b4f35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:19:24 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:58634
I20260924 15:19:24.113186 10123 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ac1cc9b7102438bb406c3ab920b4f35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:19:24 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:42490
I20260924 15:19:24.113271 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 9787
I20260924 15:19:24.114575 10229 tablet_replica.cc:330] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800: stopping tablet replica
I20260924 15:19:24.114686 10229 raft_consensus.cc:2229] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:19:24.114741 10228 tablet_replica.cc:330] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c: stopping tablet replica
I20260924 15:19:24.114794 10229 raft_consensus.cc:2258] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:19:24.114846 10228 raft_consensus.cc:2229] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:19:24.114950 10228 raft_consensus.cc:2258] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:19:24.115208 10229 ts_tablet_manager.cc:1905] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:19:24.115517 10228 ts_tablet_manager.cc:1905] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:19:24.116948 10229 ts_tablet_manager.cc:1918] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:19:24.117043 10229 log.cc:1196] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/8ac1cc9b7102438bb406c3ab920b4f35
I20260924 15:19:24.117306 10229 ts_tablet_manager.cc:1939] T 8ac1cc9b7102438bb406c3ab920b4f35 P 283029ffbace4ce1aa3be8387e25a800: Deleting consensus metadata
I20260924 15:19:24.117584 10228 ts_tablet_manager.cc:1918] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:19:24.117681 10228 log.cc:1196] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/8ac1cc9b7102438bb406c3ab920b4f35
I20260924 15:19:24.117724 9710 catalog_manager.cc:4841] TS 283029ffbace4ce1aa3be8387e25a800 (127.31.12.67:43271): tablet 8ac1cc9b7102438bb406c3ab920b4f35 (table test-table [id=96b1579e58fd47f190f1aeff765fea0b]) successfully deleted
I20260924 15:19:24.118014 10228 ts_tablet_manager.cc:1939] T 8ac1cc9b7102438bb406c3ab920b4f35 P 8125f364d96741b88c77034821be151c: Deleting consensus metadata
I20260924 15:19:24.118597 9711 catalog_manager.cc:4841] TS 8125f364d96741b88c77034821be151c (127.31.12.66:35585): tablet 8ac1cc9b7102438bb406c3ab920b4f35 (table test-table [id=96b1579e58fd47f190f1aeff765fea0b]) successfully deleted
W20260924 15:19:24.119871 9709 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.12.65:33687 (error 108)
I20260924 15:19:24.120118 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 9924
I20260924 15:19:24.120705 10199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:19:24.120197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:33687 (local address 127.0.0.1:52280)
0924 15:19:24.120636 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 15:19:24.120669 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.65:33687: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421}
W20260924 15:19:24.122048 9709 catalog_manager.cc:4584] TS ee73ffbe183f4d5eb44bea3e70da6a68 (127.31.12.65:33687): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ac1cc9b7102438bb406c3ab920b4f35: Network error: Client connection negotiation failed: client connection to 127.31.12.65:33687: connect: Connection refused (error 111)
I20260924 15:19:24.124730 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 10059
I20260924 15:19:24.129704 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 9692
2026-09-24T15:19:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1906 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:19:24 dist-test-slave-zwbb krb5kdc[10233](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:19:24 dist-test-slave-zwbb krb5kdc[10233](info): set up 2 sockets
Sep 24 15:19:24 dist-test-slave-zwbb krb5kdc[10233](info): commencing operation
krb5kdc: starting...
W20260924 15:19:25.163436 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:19:25 dist-test-slave-zwbb krb5kdc[10233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:19:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:19:25Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/rangeradmin_127.31.12.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/rangeradmin_127.31.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/rangerlookup_127.31.12.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/rangerlookup_127.31.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/HTTP_127.31.12.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/HTTP_127.31.12.84@KRBTEST.COM.keytab.
I20260924 15:19:25.324422 31793 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskqOs6Q8/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/postgres -l logfile start
2026-09-24 15:19:27.830 UTC [10264] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 15:19:27.830 UTC [10264] LOG: listening on IPv4 address "127.31.12.84", port 54469
2026-09-24 15:19:27.834 UTC [10264] LOG: listening on Unix socket "/tmp/.s.PGSQL.54469"
2026-09-24 15:19:27.869 UTC [10267] LOG: database system was shut down at 2026-09-24 15:19:26 UTC
2026-09-24 15:19:27.873 UTC [10264] LOG: database system is ready to accept connections
I20260924 15:19:28.827214 31793 mini_postgres.cc:96] Postgres bound to 54469
2026-09-24 15:19:28.834 UTC [10275] FATAL: database "slave" does not exist
127.31.12.84:54469 - accepting connections
I20260924 15:19:28.835312 31793 mini_ranger.cc:161] Starting Ranger...
I20260924 15:19:28.850432 31793 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 15:19:29.350078 31793 mini_ranger.cc:87] Created ranger Postgres database
I20260924 15:19:29.350140 31793 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ranger-admin
2026-09-24 15:19:29,784 [I] DB FLAVOR :POSTGRES
2026-09-24 15:19:29,784 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 15:19:29,784 [I] Checking connection..
2026-09-24 15:19:29,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 15:19:30,068 [I] Checking connection passed.
2026-09-24 15:19:30,069 [I] --------- Verifying version history table ---------
2026-09-24 15:19:30,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 15:19:30,302 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 15:19:30,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 15:19:30,561 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 15:19:30,561 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 15:19:30,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 15:19:30.804 UTC [10339] WARNING: there is no transaction in progress
2026-09-24 15:19:30,811 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 15:19:30,811 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 15:19:31,071 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 15:19:31,072 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 15:19:31,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 15:19:31,318 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 15:19:31,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zwbb.c.gcp-upstream.internal','N') ;"
2026-09-24 15:19:31,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 15:19:32,032 [I] Table x_portal_user does not exist in database ranger
2026-09-24 15:19:32,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 15:19:32,275 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 15:19:32,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 15:19:32,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 15:19:32,787 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 15:19:32,787 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 15:19:33.876 UTC [10453] WARNING: there is no transaction in progress
2026-09-24 15:19:33.903 UTC [10453] WARNING: there is no transaction in progress
2026-09-24 15:19:33.922 UTC [10453] WARNING: there is no transaction in progress
2026-09-24 15:19:33.941 UTC [10453] WARNING: there is no transaction in progress
2026-09-24 15:19:33.974 UTC [10453] WARNING: there is no transaction in progress
2026-09-24 15:19:33.990 UTC [10453] WARNING: there is no transaction in progress
2026-09-24 15:19:34.054 UTC [10453] WARNING: there is no transaction in progress
2026-09-24 15:19:34.060 UTC [10453] WARNING: there is no transaction in progress
2026-09-24 15:19:34.074 UTC [10453] WARNING: there is no transaction in progress
2026-09-24 15:19:34.085 UTC [10453] WARNING: there is no transaction in progress
2026-09-24 15:19:34.100 UTC [10453] WARNING: there is no transaction in progress
2026-09-24 15:19:34.128 UTC [10453] WARNING: there is no transaction in progress
2026-09-24 15:19:34.142 UTC [10453] WARNING: there is no transaction in progress
2026-09-24 15:19:34.154 UTC [10453] WARNING: there is no transaction in progress
2026-09-24 15:19:34.160 UTC [10453] WARNING: there is no transaction in progress
2026-09-24 15:19:34.166 UTC [10453] WARNING: there is no transaction in progress
2026-09-24 15:19:34,763 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 15:19:34,763 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 15:19:35,014 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 15:19:35,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 15:19:35,252 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 15:19:35,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 15:19:35,514 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 15:19:35,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 15:19:35,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 15:19:36,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zwbb.c.gcp-upstream.internal';"
2026-09-24 15:19:36,254 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 15:19:36,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 15:19:36,504 [I] DB_PATCHES have already been applied
I20260924 15:19:36.510106 31793 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ranger-admin:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskqOs6Q8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 15:19:36.510169 31793 mini_ranger.cc:193] Using host: 127.31.12.84
I20260924 15:19:36.513275 31793 mini_ranger.cc:227] Ranger admin URL: http://127.31.12.84:46625
Sep 24, 2026 3:19:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:19:36 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:19:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 3:19:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 3:19:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 3:19:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 3:19:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 15:19:38.751715 31793 mini_ranger.cc:160] Time spent starting Ranger: real 9.916s user 0.004s sys 0.007s
Sep 24, 2026 3:19:58 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 3:19:58 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 3:19:58 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 3:19:58 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 3:19:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 3:19:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 3:19:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 3:19:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 3:19:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 3:19:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 3:19:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 3:19:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 3:19:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 3:19:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 3:19:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 3:19:58 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 3:19:58 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 3:19:59 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 15:19:59.848963 31793 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/rangerkms_127.31.12.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/rangerkms_127.31.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.12.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.12.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.12.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/HTTP_127.31.12.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.12.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/HTTP_127.31.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 15:19:59 dist-test-slave-zwbb krb5kdc[10233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263199, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 15:19:59.933254 31793 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 15:19:59.942798 31793 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 15:20:01.261005 31793 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 15:20:01.261089 31793 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ranger-kms
I20260924 15:20:01.261108 31793 mini_ranger_kms.cc:227] Using postgres at 127.31.12.84:54469
2026-09-24 15:20:01,662 [I] DB FLAVOR :POSTGRES
2026-09-24 15:20:01,663 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 15:20:01,663 [I] Checking connection
2026-09-24 15:20:01,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 15:20:01,923 [I] connection success
2026-09-24 15:20:01,924 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 15:20:01,924 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 15:20:01,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 15:20:02,173 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 15:20:02,173 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 15:20:02,174 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 15:20:02,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:54469/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 15:20:02,425 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 15:20:02.941493 31793 mini_ranger_kms.cc:310] Created kms service
I20260924 15:20:03.163203 31793 mini_ranger_kms.cc:326] Created kudu user
I20260924 15:20:03.207654 31793 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 15:20:03 dist-test-slave-zwbb krb5kdc[10233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263199, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
I20260924 15:20:03.624028 31793 mini_ranger_kms.cc:384] Added ranger policy
I20260924 15:20:03.624143 31793 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ranger-kms:/tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 15:20:03.624169 31793 mini_ranger_kms.cc:240] Using host: 127.31.12.84
I20260924 15:20:03.632838 31793 mini_ranger_kms.cc:276] Ranger KMS PID: 10649
I20260924 15:20:03.632982 31793 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.12.84:47165
Sep 24, 2026 3:20:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:20:04 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:20:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 3:20:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 3:20:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 3:20:05 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 15:20:05 dist-test-slave-zwbb krb5kdc[10233](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790263205, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.12.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:20:05.951390 31793 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.018s user 0.000s sys 0.215s
I20260924 15:20:05.951506 31793 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 15:20:05.951596 31793 mini_ranger_kms.cc:401] 127.31.12.84:47165/kms/v1/keys
Sep 24 15:20:06 dist-test-slave-zwbb krb5kdc[10233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263205, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.12.84@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
Sep 24 15:20:06 dist-test-slave-zwbb krb5kdc[10233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263205, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.12.84@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
Sep 24 15:20:09 dist-test-slave-zwbb krb5kdc[10233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:20:09.179697 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:37511
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:45837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:37511
--ranger_config_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.12.84:47165/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:20:09.259660 10700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:20:09.259846 10700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:20:09.259866 10700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:20:09.260982 10700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:20:09.261022 10700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:20:09.261036 10700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:20:09.261049 10700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:20:09.261061 10700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:20:09.262598 10700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:45837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.12.84:47165/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:37511
--ranger_config_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:37511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:20:09.262825 10700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:20:09.263032 10700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:20:09.267817 10706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:09.267796 10705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:20:09.267953 10700 server_base.cc:1034] running on GCE node
W20260924 15:20:09.267962 10708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:20:09.268397 10700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:20:09.268731 10700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:20:09.269879 10700 hybrid_clock.cc:648] HybridClock initialized: now 1790263209269854 us; error 42 us; skew 500 ppm
Sep 24 15:20:09 dist-test-slave-zwbb krb5kdc[10233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263209, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:20:09.275892 10700 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:20:09.276408 10700 webserver.cc:460] Webserver started at http://127.31.12.126:32783/ using document root <none> and password file <none>
I20260924 15:20:09.276648 10700 fs_manager.cc:359] Metadata directory not provided
I20260924 15:20:09.276708 10700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:20:09.276868 10700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:20:09 dist-test-slave-zwbb krb5kdc[10233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263209, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
I20260924 15:20:09.323509 10700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "999cf1e1730e408db75b5e83e8cd1118"
format_stamp: "Formatted at 2026-09-24 15:20:09 on dist-test-slave-zwbb"
server_key: "ea17543d306ce11ae226754406ee7e38"
server_key_iv: "03fb0382843aa7db26ed566db2b84b3b"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:09.323941 10700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "999cf1e1730e408db75b5e83e8cd1118"
format_stamp: "Formatted at 2026-09-24 15:20:09 on dist-test-slave-zwbb"
server_key: "ea17543d306ce11ae226754406ee7e38"
server_key_iv: "03fb0382843aa7db26ed566db2b84b3b"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:09.325729 10700 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 15:20:09.337101 10715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:09.337358 10700 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:20:09.337518 10700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "999cf1e1730e408db75b5e83e8cd1118"
format_stamp: "Formatted at 2026-09-24 15:20:09 on dist-test-slave-zwbb"
server_key: "ea17543d306ce11ae226754406ee7e38"
server_key_iv: "03fb0382843aa7db26ed566db2b84b3b"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:09.337646 10700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:20:09.353749 10700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:20:09.355638 10700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:20:09.355794 10700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:20:09.360622 10700 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:37511
I20260924 15:20:09.360702 10768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:37511 every 8 connection(s)
I20260924 15:20:09.361124 10700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:20:09.361845 10769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:20:09.364049 10769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118: Bootstrap starting.
I20260924 15:20:09.364674 10769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118: Neither blocks nor log segments found. Creating new log.
I20260924 15:20:09.364856 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 10700
I20260924 15:20:09.364995 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:20:09.365082 10769 log.cc:824] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118: Log is configured to *not* fsync() on all Append() calls
I20260924 15:20:09.365199 31793 external_mini_cluster.cc:1428] Setting key c03d7e171a46cb30c80c5f6e2cc45412
I20260924 15:20:09.365936 10769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118: No bootstrap required, opened a new log
I20260924 15:20:09.368046 10769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999cf1e1730e408db75b5e83e8cd1118" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37511 } }
I20260924 15:20:09.368202 10769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:20:09.368263 10769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 999cf1e1730e408db75b5e83e8cd1118, State: Initialized, Role: FOLLOWER
I20260924 15:20:09.368405 10769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999cf1e1730e408db75b5e83e8cd1118" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37511 } }
I20260924 15:20:09.368508 10769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:20:09.368562 10769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:20:09.368633 10769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:20:09.369130 10769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999cf1e1730e408db75b5e83e8cd1118" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37511 } }
I20260924 15:20:09.369246 10769 leader_election.cc:302] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 999cf1e1730e408db75b5e83e8cd1118; no voters:
I20260924 15:20:09.369393 10769 leader_election.cc:288] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:20:09.369422 10774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:20:09.369649 10774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118 [term 1 LEADER]: Becoming Leader. State: Replica: 999cf1e1730e408db75b5e83e8cd1118, State: Running, Role: LEADER
I20260924 15:20:09.369702 10769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:20:09.369788 10774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999cf1e1730e408db75b5e83e8cd1118" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37511 } }
I20260924 15:20:09.370157 10774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "999cf1e1730e408db75b5e83e8cd1118" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999cf1e1730e408db75b5e83e8cd1118" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37511 } } }
I20260924 15:20:09.370328 10774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118 [sys.catalog]: This master's current role is: LEADER
I20260924 15:20:09.370522 10774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 999cf1e1730e408db75b5e83e8cd1118. Latest consensus state: current_term: 1 leader_uuid: "999cf1e1730e408db75b5e83e8cd1118" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "999cf1e1730e408db75b5e83e8cd1118" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 37511 } } }
I20260924 15:20:09.370625 10774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118 [sys.catalog]: This master's current role is: LEADER
I20260924 15:20:09.370795 10783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:20:09.371188 10769 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 15:20:09.371367 10783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 15:20:09 dist-test-slave-zwbb krb5kdc[10233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263209, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:20:09.374248 10791 mvcc.cc:205] Tried to move back new op lower bound from 7332918105588834304 to 7332918105579175936. Current Snapshot: MvccSnapshot[applied={T|T < 7332918105588834304}]
I20260924 15:20:09.381618 10783 catalog_manager.cc:1347] Generated new cluster ID: 95019d5c9929465b841695530a27fc75
I20260924 15:20:09.381682 10783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:20:09.383603 10773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:09.365730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45000 (local address 127.31.12.126:37511)
0924 15:20:09.365886 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:09.365896 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:09.365931 (+ 35us) server_negotiation.cc:407] Connection header received
0924 15:20:09.366474 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:09.366483 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:09.366570 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:09.366725 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:20:09.367475 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:09.369137 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:09.370230 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:09.370808 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:09.370927 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:09.373773 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:20:09.373777 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:20:09.373777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:20:09.373789 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:20:09.377881 (+ 4092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:09.381814 (+ 3933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:09.381817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:09.381818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:09.381934 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:09.382179 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:09.382180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:09.382181 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:09.382519 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:20:09.383139 (+ 620us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:09.383212 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 15:20:09.383361 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260924 15:20:09.389356 10783 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:20:09.390206 10783 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:20:09.399071 10783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118: Generated new TSK 0
I20260924 15:20:09.399323 10783 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
I20260924 15:20:10.141491 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.12.84:47165/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:37511
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:45837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:20:10.228935 10814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:20:10.229163 10814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:20:10.229200 10814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:20:10.230423 10814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:20:10.230494 10814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:20:10.230540 10814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:20:10.232087 10814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:45837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.12.84:47165/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:37511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:20:10.232347 10814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:20:10.232581 10814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:20:10.237665 10819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:10.237785 10822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:10.237682 10820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:20:10.238051 10814 server_base.cc:1034] running on GCE node
I20260924 15:20:10.238271 10814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:20:10.238504 10814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:20:10.239660 10814 hybrid_clock.cc:648] HybridClock initialized: now 1790263210239629 us; error 48 us; skew 500 ppm
Sep 24 15:20:10 dist-test-slave-zwbb krb5kdc[10233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263210, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:20:10.242271 10814 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:20:10.242759 10814 webserver.cc:460] Webserver started at http://127.31.12.65:35495/ using document root <none> and password file <none>
I20260924 15:20:10.243006 10814 fs_manager.cc:359] Metadata directory not provided
I20260924 15:20:10.243073 10814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:20:10.243203 10814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:20:10 dist-test-slave-zwbb krb5kdc[10233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263210, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
I20260924 15:20:10.258327 10814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "e6318bd2eb89456fbf27198154448faf"
format_stamp: "Formatted at 2026-09-24 15:20:10 on dist-test-slave-zwbb"
server_key: "b8278daaccccecfc170c1ef3cddb604d"
server_key_iv: "45ea808ab9b4d6e4050b95c255d40dc7"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:10.258863 10814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "e6318bd2eb89456fbf27198154448faf"
format_stamp: "Formatted at 2026-09-24 15:20:10 on dist-test-slave-zwbb"
server_key: "b8278daaccccecfc170c1ef3cddb604d"
server_key_iv: "45ea808ab9b4d6e4050b95c255d40dc7"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:10.260713 10814 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 15:20:10.277946 10829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:10.278177 10814 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 15:20:10.278275 10814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "e6318bd2eb89456fbf27198154448faf"
format_stamp: "Formatted at 2026-09-24 15:20:10 on dist-test-slave-zwbb"
server_key: "b8278daaccccecfc170c1ef3cddb604d"
server_key_iv: "45ea808ab9b4d6e4050b95c255d40dc7"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:10.278368 10814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:20:10.315017 10814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:20:10.317363 10814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:20:10.317576 10814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:20:10.317837 10814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:20:10.318228 10814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:20:10.318282 10814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:10.318338 10814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:20:10.318394 10814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:10.325740 10814 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:34851
I20260924 15:20:10.325860 10945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:34851 every 8 connection(s)
I20260924 15:20:10.326296 10814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:20:10.331043 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 10814
I20260924 15:20:10.331151 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:20:10.331351 31793 external_mini_cluster.cc:1428] Setting key 920da780e6e6c6d63d2634d9e7f14a67
Sep 24 15:20:10 dist-test-slave-zwbb krb5kdc[10233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263210, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:20:10.336655 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:10.327338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:39263 (local address 127.31.12.126:37511)
0924 15:20:10.327478 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:10.327480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:10.327682 (+ 202us) server_negotiation.cc:407] Connection header received
0924 15:20:10.328195 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:10.328196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:10.328210 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:10.328274 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:20:10.328843 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:10.329358 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:10.330398 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:10.330601 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:10.330706 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:10.333391 (+ 2685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:20:10.333392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:20:10.333392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:20:10.333399 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:20:10.335275 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:10.335690 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:10.335691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:10.335691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:10.335756 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:10.336028 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:10.336029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:10.336029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:10.336145 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:20:10.336218 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:10.336482 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 15:20:10.336564 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260924 15:20:10.336843 10949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:10.327333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37511 (local address 127.31.12.65:39263)
0924 15:20:10.327574 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:10.327603 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 15:20:10.328096 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:20:10.328355 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:20:10.328359 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:20:10.328548 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:20:10.328750 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:10.328755 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:10.329495 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:10.329496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:10.330294 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:10.330296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:10.330713 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:10.330714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:10.330888 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:10.331530 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:20:10.331541 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:20:10.333291 (+ 1750us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:20:10.335374 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:20:10.335380 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:20:10.335383 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:20:10.335599 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:20:10.335825 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:20:10.335826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:20:10.335826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:20:10.335946 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:20:10.336210 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:20:10.336216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:20:10.336345 (+ 129us) client_negotiation.cc:769] Sending connection context
0924 15:20:10.336454 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 15:20:10.336594 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
I20260924 15:20:10.337430 10946 heartbeater.cc:347] Connected to a master server at 127.31.12.126:37511
I20260924 15:20:10.337543 10946 heartbeater.cc:464] Registering TS with master...
I20260924 15:20:10.337711 10946 heartbeater.cc:511] Master 127.31.12.126:37511 requested a full tablet report, sending...
I20260924 15:20:10.338271 10732 ts_manager.cc:194] Registered new tserver with Master: e6318bd2eb89456fbf27198154448faf (127.31.12.65:34851)
I20260924 15:20:10.338872 10732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:39263
WARNING: no policy specified for kudu/127.31.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
I20260924 15:20:10.365231 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.12.84:47165/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:37511
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:45837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:20:10.475065 10952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:20:10.475328 10952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:20:10.475365 10952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:20:10.476979 10952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:20:10.477039 10952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:20:10.477079 10952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:20:10.479216 10952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:45837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.12.84:47165/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:37511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:20:10.479475 10952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
Sep 24 15:20:10 dist-test-slave-zwbb krb5kdc[10233](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790263210, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:20:10.479764 10952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:20:10.486188 10959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:10.486346 10960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:10.486474 10962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:20:10.487718 10952 server_base.cc:1034] running on GCE node
I20260924 15:20:10.487926 10952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:20:10.488186 10952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:20:10.491448 10952 hybrid_clock.cc:648] HybridClock initialized: now 1790263210490926 us; error 536 us; skew 500 ppm
Sep 24 15:20:10 dist-test-slave-zwbb krb5kdc[10233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263210, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:20:10.494611 10952 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:20:10.495110 10952 webserver.cc:460] Webserver started at http://127.31.12.66:40631/ using document root <none> and password file <none>
I20260924 15:20:10.495335 10952 fs_manager.cc:359] Metadata directory not provided
I20260924 15:20:10.495381 10952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:20:10.495499 10952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:20:10 dist-test-slave-zwbb krb5kdc[10233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263210, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
I20260924 15:20:10.521948 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "6b6f8b2ee66e41d48259916d7f334712"
format_stamp: "Formatted at 2026-09-24 15:20:10 on dist-test-slave-zwbb"
server_key: "34d3e626e31725e420e150327c2b88fe"
server_key_iv: "4501b58dda88c101aa2b56decc680c39"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:10.522444 10952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "6b6f8b2ee66e41d48259916d7f334712"
format_stamp: "Formatted at 2026-09-24 15:20:10 on dist-test-slave-zwbb"
server_key: "34d3e626e31725e420e150327c2b88fe"
server_key_iv: "4501b58dda88c101aa2b56decc680c39"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:10.525117 10952 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 15:20:10.551321 10969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:10.551620 10952 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260924 15:20:10.551728 10952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "6b6f8b2ee66e41d48259916d7f334712"
format_stamp: "Formatted at 2026-09-24 15:20:10 on dist-test-slave-zwbb"
server_key: "34d3e626e31725e420e150327c2b88fe"
server_key_iv: "4501b58dda88c101aa2b56decc680c39"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:10.551832 10952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:20:10.598565 10952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:20:10.601066 10952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:20:10.601336 10952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:20:10.601641 10952 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:20:10.602083 10952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:20:10.602178 10952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:10.602254 10952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:20:10.602315 10952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:10.608722 10952 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:34573
I20260924 15:20:10.609230 10952 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:20:10.611445 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 10952
I20260924 15:20:10.611539 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:20:10.611732 31793 external_mini_cluster.cc:1428] Setting key 1ef9cc0cc93d0fce0acb7a185601a2d4
WARNING: no policy specified for kudu/127.31.12.67@KRBTEST.COM; defaulting to no policy
I20260924 15:20:10.622577 11086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:34573 every 8 connection(s)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.67@KRBTEST.COM" created.
Sep 24 15:20:10 dist-test-slave-zwbb krb5kdc[10233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263210, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:20:10.638721 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:10.623644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:39263 (local address 127.31.12.126:37511)
0924 15:20:10.623697 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:10.623700 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:10.624022 (+ 322us) server_negotiation.cc:407] Connection header received
0924 15:20:10.624919 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:10.624921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:10.624939 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:10.624983 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:20:10.628074 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:10.628859 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:10.631518 (+ 2659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:10.631813 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:10.631862 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:10.635185 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:20:10.635188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:20:10.635188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:20:10.635196 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:20:10.636991 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:10.637545 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:10.637547 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:10.637547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:10.637611 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:10.637942 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:10.637943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:10.637944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:10.638083 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:20:10.638141 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:10.638526 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 15:20:10.638620 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:20:10.639473 11090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:10.623334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37511 (local address 127.31.12.66:39263)
0924 15:20:10.623863 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:10.623915 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 15:20:10.624753 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:20:10.627383 (+ 2630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:20:10.627388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:20:10.627634 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:20:10.627935 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:10.627942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:10.629965 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:10.629966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:10.631405 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:10.631408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:10.632286 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:10.632287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:10.632453 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:10.633092 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:20:10.633103 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:20:10.635066 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:20:10.637090 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:20:10.637097 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:20:10.637100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:20:10.637408 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:20:10.637710 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:20:10.637711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:20:10.637711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:20:10.637853 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:20:10.638177 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:20:10.638184 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:20:10.638347 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 15:20:10.639040 (+ 693us) client_negotiation.cc:240] Negotiation successful
0924 15:20:10.639217 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":100,"threads_started":1}
I20260924 15:20:10.640192 11087 heartbeater.cc:347] Connected to a master server at 127.31.12.126:37511
I20260924 15:20:10.640293 11087 heartbeater.cc:464] Registering TS with master...
I20260924 15:20:10.640501 11087 heartbeater.cc:511] Master 127.31.12.126:37511 requested a full tablet report, sending...
I20260924 15:20:10.640933 10732 ts_manager.cc:194] Registered new tserver with Master: 6b6f8b2ee66e41d48259916d7f334712 (127.31.12.66:34573)
I20260924 15:20:10.641405 10732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:39263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
I20260924 15:20:10.661755 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.12.84:47165/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:37511
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:45837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:20:10.847965 11092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:20:10.848316 11092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:20:10.848407 11092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:20:10.850176 11092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:20:10.850405 11092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:20:10.850510 11092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:20:10.852700 11092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:45837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.12.84:47165/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:37511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:20:10.853423 11092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:20:10.853801 11092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:20:10.860744 11100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:10.861418 11097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:20:10.861723 11092 server_base.cc:1034] running on GCE node
W20260924 15:20:10.861804 11098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:20:10.862107 11092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:20:10.862397 11092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:20:10.863601 11092 hybrid_clock.cc:648] HybridClock initialized: now 1790263210863581 us; error 34 us; skew 500 ppm
Sep 24 15:20:10 dist-test-slave-zwbb krb5kdc[10233](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263210, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:20:10.866850 11092 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:20:10.867348 11092 webserver.cc:460] Webserver started at http://127.31.12.67:46623/ using document root <none> and password file <none>
I20260924 15:20:10.867631 11092 fs_manager.cc:359] Metadata directory not provided
I20260924 15:20:10.867683 11092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:20:10.867808 11092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:20:10 dist-test-slave-zwbb krb5kdc[10233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263210, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
I20260924 15:20:10.884140 11092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "766a0be7b2c04fecb0f580f3636fee92"
format_stamp: "Formatted at 2026-09-24 15:20:10 on dist-test-slave-zwbb"
server_key: "5ca0900fe553c5b2f4d49a543074d60b"
server_key_iv: "326371812daaed4b5cff0359db1712ce"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:10.884704 11092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "766a0be7b2c04fecb0f580f3636fee92"
format_stamp: "Formatted at 2026-09-24 15:20:10 on dist-test-slave-zwbb"
server_key: "5ca0900fe553c5b2f4d49a543074d60b"
server_key_iv: "326371812daaed4b5cff0359db1712ce"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:10.889847 11092 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.002s
I20260924 15:20:10.913172 11108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:10.917688 11092 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.000s sys 0.001s
I20260924 15:20:10.917853 11092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "766a0be7b2c04fecb0f580f3636fee92"
format_stamp: "Formatted at 2026-09-24 15:20:10 on dist-test-slave-zwbb"
server_key: "5ca0900fe553c5b2f4d49a543074d60b"
server_key_iv: "326371812daaed4b5cff0359db1712ce"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:10.917991 11092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:20:10.939118 11092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:20:10.941823 11092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:20:10.942057 11092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:20:10.942435 11092 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:20:10.942858 11092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:20:10.942952 11092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:10.943032 11092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:20:10.943094 11092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:10.950671 11092 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:38831
I20260924 15:20:10.951184 11092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:20:10.953521 11222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:38831 every 8 connection(s)
I20260924 15:20:10.960665 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 11092
I20260924 15:20:10.960773 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:20:10.960985 31793 external_mini_cluster.cc:1428] Setting key 768aba25cf79ef98defeb07e1a5efc21
Sep 24 15:20:10 dist-test-slave-zwbb krb5kdc[10233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263210, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:20:10.966468 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:10.954648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:35609 (local address 127.31.12.126:37511)
0924 15:20:10.954694 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:10.954697 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:10.955011 (+ 314us) server_negotiation.cc:407] Connection header received
0924 15:20:10.955763 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:10.955764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:10.955777 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:10.955809 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:20:10.956685 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:10.957526 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:10.959097 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:10.959402 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:10.959448 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:10.963044 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:20:10.963045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:20:10.963046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:20:10.963053 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:20:10.964793 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:10.965315 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:10.965316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:10.965317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:10.965377 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:10.965714 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:10.965715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:10.965715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:10.965848 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:20:10.965893 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:10.966281 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 15:20:10.966365 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:20:10.967236 11225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:10.954343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:37511 (local address 127.31.12.67:35609)
0924 15:20:10.954877 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:10.954920 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 15:20:10.955596 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:20:10.956038 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:20:10.956042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:20:10.956285 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:20:10.956575 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:10.956581 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:10.957665 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:10.957666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:10.958946 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:10.958948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:10.959541 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:10.959542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:10.959688 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:10.960297 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:20:10.960307 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:20:10.962936 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:20:10.964911 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:20:10.964919 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:20:10.964921 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:20:10.965225 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:20:10.965491 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:20:10.965492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:20:10.965492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:20:10.965633 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:20:10.965947 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:20:10.965953 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:20:10.966114 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 15:20:10.966828 (+ 714us) client_negotiation.cc:240] Negotiation successful
0924 15:20:10.966996 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":109,"threads_started":1}
I20260924 15:20:10.967906 11223 heartbeater.cc:347] Connected to a master server at 127.31.12.126:37511
I20260924 15:20:10.968001 11223 heartbeater.cc:464] Registering TS with master...
I20260924 15:20:10.968202 11223 heartbeater.cc:511] Master 127.31.12.126:37511 requested a full tablet report, sending...
I20260924 15:20:10.968636 10732 ts_manager.cc:194] Registered new tserver with Master: 766a0be7b2c04fecb0f580f3636fee92 (127.31.12.67:38831)
I20260924 15:20:10.969112 10732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:35609
I20260924 15:20:10.973358 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:20:10.983139 10948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:10.975169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60798 (local address 127.31.12.126:37511)
0924 15:20:10.975211 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:10.975213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:10.975505 (+ 292us) server_negotiation.cc:407] Connection header received
0924 15:20:10.975602 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:10.975603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:10.975613 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:10.975643 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:20:10.976429 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:10.977140 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:10.978207 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:10.978472 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:10.978518 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:10.980103 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:20:10.980105 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:20:10.980105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:20:10.980112 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:20:10.981816 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:10.982228 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:10.982229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:10.982230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:10.982285 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:10.982497 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:10.982498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:10.982498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:10.982625 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:20:10.982696 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:10.982960 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 15:20:10.983039 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:20:10.985432 10732 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:20:10.987807 10732 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:20:11.000800 11237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:10.991426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:38831 (local address 127.0.0.1:35090)
0924 15:20:10.993013 (+ 1587us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:10.993057 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 15:20:10.993124 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:20:10.994190 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:20:10.994191 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:20:10.994192 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:20:10.994227 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:10.994228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:10.996030 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:10.996031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:10.999380 (+ 3349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:10.999383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:11.000476 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:11.000478 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:11.000561 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:11.000576 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 15:20:11.000649 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 15:20:11.000705 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1555,"thread_start_us":80,"threads_started":1}
I20260924 15:20:11.006775 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:10.992249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.31.12.67:38831)
0924 15:20:10.993398 (+ 1149us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:10.993403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:10.993414 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:20:10.993497 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:10.993499 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:10.993516 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:10.993593 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:20:10.994857 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:10.995932 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:10.999536 (+ 3604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:11.000278 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:11.000353 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:11.000438 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:11.006614 (+ 6176us) server_negotiation.cc:299] Negotiation successful
0924 15:20:11.006677 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1101,"thread_start_us":72,"threads_started":1}
I20260924 15:20:11.008174 11151 tablet_service.cc:1467] Processing CreateTablet for tablet bfe967d9462d474b9a4a47d0299042da (DEFAULT_TABLE table=test-table [id=c2e2a9b38d804bd588d8fb43b3cac7e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:20:11.008582 11151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfe967d9462d474b9a4a47d0299042da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:20:11.019394 11236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:10.991825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:34851 (local address 127.0.0.1:47600)
0924 15:20:10.998752 (+ 6927us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:10.998763 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:20:10.998823 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:20:11.013163 (+ 14340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:20:11.013164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:20:11.013167 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:20:11.013225 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:11.013227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:11.016377 (+ 3150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:11.016378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:11.017555 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:11.017559 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:11.019130 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:11.019131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:11.019204 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:11.019211 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 15:20:11.019250 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 15:20:11.019300 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6908,"thread_start_us":63,"threads_started":1}
I20260924 15:20:11.019762 11241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:10.992602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47600 (local address 127.31.12.65:34851)
0924 15:20:10.993993 (+ 1391us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:10.993997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:11.012961 (+ 18964us) server_negotiation.cc:407] Connection header received
0924 15:20:11.013012 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:11.013014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:11.013041 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:11.013303 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:20:11.013373 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:11.014528 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:11.017667 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:11.018474 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:11.019531 (+ 1057us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:11.019626 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:11.019654 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 15:20:11.019697 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1360,"thread_start_us":46,"threads_started":1}
I20260924 15:20:11.020538 10879 tablet_service.cc:1467] Processing CreateTablet for tablet bfe967d9462d474b9a4a47d0299042da (DEFAULT_TABLE table=test-table [id=c2e2a9b38d804bd588d8fb43b3cac7e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:20:11.020919 10879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfe967d9462d474b9a4a47d0299042da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:20:11.021245 11242 tablet_bootstrap.cc:492] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92: Bootstrap starting.
I20260924 15:20:11.022395 11242 tablet_bootstrap.cc:654] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92: Neither blocks nor log segments found. Creating new log.
I20260924 15:20:11.022727 11242 log.cc:824] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92: Log is configured to *not* fsync() on all Append() calls
I20260924 15:20:11.025158 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:10.991653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:34573 (local address 127.0.0.1:48400)
0924 15:20:10.993232 (+ 1579us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:10.993240 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:20:10.993273 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:20:10.994041 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:20:10.994044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:20:10.994047 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:20:10.994116 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:10.994117 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:11.019259 (+ 25142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:11.019260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:11.023750 (+ 4490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:11.023754 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:11.024910 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:11.024911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:11.024982 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:11.024987 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:20:11.025039 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 15:20:11.025081 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1563,"mutex_wait_us":22,"thread_start_us":63,"threads_started":1}
I20260924 15:20:11.025573 11240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:10.992425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48400 (local address 127.31.12.66:34573)
0924 15:20:10.993734 (+ 1309us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:10.993738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:10.993747 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:20:10.993783 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:10.993785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:10.993799 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:10.993873 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:20:10.994298 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:11.016237 (+ 21939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:11.023905 (+ 7668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:11.024698 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:11.024770 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:11.024869 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:11.025425 (+ 556us) server_negotiation.cc:299] Negotiation successful
0924 15:20:11.025495 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1275,"thread_start_us":87,"threads_started":1}
I20260924 15:20:11.026758 11020 tablet_service.cc:1467] Processing CreateTablet for tablet bfe967d9462d474b9a4a47d0299042da (DEFAULT_TABLE table=test-table [id=c2e2a9b38d804bd588d8fb43b3cac7e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:20:11.027123 11020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfe967d9462d474b9a4a47d0299042da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:20:11.027477 11242 tablet_bootstrap.cc:492] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92: No bootstrap required, opened a new log
I20260924 15:20:11.027557 11242 ts_tablet_manager.cc:1397] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260924 15:20:11.029673 11242 raft_consensus.cc:348] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766a0be7b2c04fecb0f580f3636fee92" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38831 } } peers { permanent_uuid: "6b6f8b2ee66e41d48259916d7f334712" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 34573 } } peers { permanent_uuid: "e6318bd2eb89456fbf27198154448faf" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34851 } }
I20260924 15:20:11.029863 11242 raft_consensus.cc:374] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:20:11.029906 11242 raft_consensus.cc:729] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 766a0be7b2c04fecb0f580f3636fee92, State: Initialized, Role: FOLLOWER
I20260924 15:20:11.030031 11242 consensus_queue.cc:260] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766a0be7b2c04fecb0f580f3636fee92" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38831 } } peers { permanent_uuid: "6b6f8b2ee66e41d48259916d7f334712" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 34573 } } peers { permanent_uuid: "e6318bd2eb89456fbf27198154448faf" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34851 } }
I20260924 15:20:11.030308 11242 ts_tablet_manager.cc:1428] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:20:11.030439 11223 heartbeater.cc:503] Master 127.31.12.126:37511 was elected leader, sending a full tablet report...
Sep 24 15:20:11 dist-test-slave-zwbb krb5kdc[10233](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263210, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
I20260924 15:20:11.041638 11246 tablet_bootstrap.cc:492] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712: Bootstrap starting.
I20260924 15:20:11.042740 11246 tablet_bootstrap.cc:654] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712: Neither blocks nor log segments found. Creating new log.
I20260924 15:20:11.043087 11246 log.cc:824] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712: Log is configured to *not* fsync() on all Append() calls
I20260924 15:20:11.043466 11244 tablet_bootstrap.cc:492] T bfe967d9462d474b9a4a47d0299042da P e6318bd2eb89456fbf27198154448faf: Bootstrap starting.
I20260924 15:20:11.044212 11244 tablet_bootstrap.cc:654] T bfe967d9462d474b9a4a47d0299042da P e6318bd2eb89456fbf27198154448faf: Neither blocks nor log segments found. Creating new log.
I20260924 15:20:11.044525 11244 log.cc:824] T bfe967d9462d474b9a4a47d0299042da P e6318bd2eb89456fbf27198154448faf: Log is configured to *not* fsync() on all Append() calls
I20260924 15:20:11.046972 11246 tablet_bootstrap.cc:492] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712: No bootstrap required, opened a new log
I20260924 15:20:11.047058 11246 ts_tablet_manager.cc:1397] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260924 15:20:11.049115 11246 raft_consensus.cc:348] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766a0be7b2c04fecb0f580f3636fee92" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38831 } } peers { permanent_uuid: "6b6f8b2ee66e41d48259916d7f334712" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 34573 } } peers { permanent_uuid: "e6318bd2eb89456fbf27198154448faf" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34851 } }
I20260924 15:20:11.049302 11246 raft_consensus.cc:374] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:20:11.049338 11246 raft_consensus.cc:729] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b6f8b2ee66e41d48259916d7f334712, State: Initialized, Role: FOLLOWER
I20260924 15:20:11.049513 11246 consensus_queue.cc:260] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766a0be7b2c04fecb0f580f3636fee92" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38831 } } peers { permanent_uuid: "6b6f8b2ee66e41d48259916d7f334712" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 34573 } } peers { permanent_uuid: "e6318bd2eb89456fbf27198154448faf" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34851 } }
I20260924 15:20:11.049762 11246 ts_tablet_manager.cc:1428] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:20:11.049875 11087 heartbeater.cc:503] Master 127.31.12.126:37511 was elected leader, sending a full tablet report...
I20260924 15:20:11.051205 11244 tablet_bootstrap.cc:492] T bfe967d9462d474b9a4a47d0299042da P e6318bd2eb89456fbf27198154448faf: No bootstrap required, opened a new log
I20260924 15:20:11.051523 11244 ts_tablet_manager.cc:1397] T bfe967d9462d474b9a4a47d0299042da P e6318bd2eb89456fbf27198154448faf: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260924 15:20:11.054090 11244 raft_consensus.cc:348] T bfe967d9462d474b9a4a47d0299042da P e6318bd2eb89456fbf27198154448faf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766a0be7b2c04fecb0f580f3636fee92" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38831 } } peers { permanent_uuid: "6b6f8b2ee66e41d48259916d7f334712" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 34573 } } peers { permanent_uuid: "e6318bd2eb89456fbf27198154448faf" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34851 } }
I20260924 15:20:11.054265 11244 raft_consensus.cc:374] T bfe967d9462d474b9a4a47d0299042da P e6318bd2eb89456fbf27198154448faf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:20:11.054301 11244 raft_consensus.cc:729] T bfe967d9462d474b9a4a47d0299042da P e6318bd2eb89456fbf27198154448faf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6318bd2eb89456fbf27198154448faf, State: Initialized, Role: FOLLOWER
I20260924 15:20:11.054437 11244 consensus_queue.cc:260] T bfe967d9462d474b9a4a47d0299042da P e6318bd2eb89456fbf27198154448faf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766a0be7b2c04fecb0f580f3636fee92" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38831 } } peers { permanent_uuid: "6b6f8b2ee66e41d48259916d7f334712" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 34573 } } peers { permanent_uuid: "e6318bd2eb89456fbf27198154448faf" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34851 } }
I20260924 15:20:11.054708 11244 ts_tablet_manager.cc:1428] T bfe967d9462d474b9a4a47d0299042da P e6318bd2eb89456fbf27198154448faf: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 15:20:11.054818 10946 heartbeater.cc:503] Master 127.31.12.126:37511 was elected leader, sending a full tablet report...
W20260924 15:20:11.077418 10947 tablet.cc:2369] T bfe967d9462d474b9a4a47d0299042da P e6318bd2eb89456fbf27198154448faf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:20:11.123479 11088 tablet.cc:2369] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:20:11.168630 11249 raft_consensus.cc:482] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:20:11.168789 11249 raft_consensus.cc:504] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766a0be7b2c04fecb0f580f3636fee92" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38831 } } peers { permanent_uuid: "6b6f8b2ee66e41d48259916d7f334712" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 34573 } } peers { permanent_uuid: "e6318bd2eb89456fbf27198154448faf" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34851 } }
I20260924 15:20:11.169202 11249 leader_election.cc:288] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 766a0be7b2c04fecb0f580f3636fee92 (127.31.12.67:38831), e6318bd2eb89456fbf27198154448faf (127.31.12.65:34851)
I20260924 15:20:11.173266 11241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:11.169610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:41097 (local address 127.31.12.65:34851)
0924 15:20:11.169654 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:11.169656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:11.169839 (+ 183us) server_negotiation.cc:407] Connection header received
0924 15:20:11.170064 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:11.170066 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:11.170076 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:11.170108 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:20:11.170321 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:11.171096 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:11.172383 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:11.172910 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:11.172966 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:11.172975 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:11.173163 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 15:20:11.173201 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:20:11.173332 11239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:11.169553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:56199 (local address 127.31.12.67:38831)
0924 15:20:11.169597 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:11.169599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:11.169706 (+ 107us) server_negotiation.cc:407] Connection header received
0924 15:20:11.169810 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:11.169811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:11.169821 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:11.169850 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:20:11.170144 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:11.170902 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:11.172388 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:11.172952 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:11.173001 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:11.173009 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:11.173227 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 15:20:11.173266 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:20:11.173621 11251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:11.169396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:38831 (local address 127.31.12.66:56199)
0924 15:20:11.169635 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:11.169648 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:20:11.169735 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:20:11.169968 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:20:11.169969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:20:11.169971 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:20:11.170045 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:11.170065 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:11.171028 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:11.171028 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:11.172272 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:11.172275 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:11.173069 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:11.173070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:11.173149 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:11.173152 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:20:11.173466 (+ 314us) client_negotiation.cc:240] Negotiation successful
0924 15:20:11.173544 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":311,"threads_started":1}
I20260924 15:20:11.173866 11252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:11.169412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:34851 (local address 127.31.12.66:41097)
0924 15:20:11.169773 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:11.169783 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:20:11.169866 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:20:11.170182 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:20:11.170183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:20:11.170185 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:20:11.170248 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:11.170250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:11.171222 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:11.171222 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:11.172286 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:11.172288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:11.173031 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:11.173032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:11.173093 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:11.173096 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:20:11.173632 (+ 536us) client_negotiation.cc:240] Negotiation successful
0924 15:20:11.173795 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":98,"threads_started":1}
I20260924 15:20:11.173976 11175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfe967d9462d474b9a4a47d0299042da" candidate_uuid: "6b6f8b2ee66e41d48259916d7f334712" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "766a0be7b2c04fecb0f580f3636fee92" is_pre_election: true
I20260924 15:20:11.174137 11175 raft_consensus.cc:2454] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b6f8b2ee66e41d48259916d7f334712 in term 0.
I20260924 15:20:11.174150 10880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfe967d9462d474b9a4a47d0299042da" candidate_uuid: "6b6f8b2ee66e41d48259916d7f334712" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6318bd2eb89456fbf27198154448faf" is_pre_election: true
I20260924 15:20:11.174280 10880 raft_consensus.cc:2454] T bfe967d9462d474b9a4a47d0299042da P e6318bd2eb89456fbf27198154448faf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b6f8b2ee66e41d48259916d7f334712 in term 0.
I20260924 15:20:11.174432 10970 leader_election.cc:302] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b6f8b2ee66e41d48259916d7f334712, 766a0be7b2c04fecb0f580f3636fee92; no voters:
I20260924 15:20:11.174577 11249 raft_consensus.cc:2781] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:20:11.174628 11249 raft_consensus.cc:482] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:20:11.174649 11249 raft_consensus.cc:3037] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:20:11.175521 11249 raft_consensus.cc:504] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766a0be7b2c04fecb0f580f3636fee92" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38831 } } peers { permanent_uuid: "6b6f8b2ee66e41d48259916d7f334712" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 34573 } } peers { permanent_uuid: "e6318bd2eb89456fbf27198154448faf" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34851 } }
I20260924 15:20:11.175643 11249 leader_election.cc:288] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [CANDIDATE]: Term 1 election: Requested vote from peers 766a0be7b2c04fecb0f580f3636fee92 (127.31.12.67:38831), e6318bd2eb89456fbf27198154448faf (127.31.12.65:34851)
I20260924 15:20:11.175853 10880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfe967d9462d474b9a4a47d0299042da" candidate_uuid: "6b6f8b2ee66e41d48259916d7f334712" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6318bd2eb89456fbf27198154448faf"
I20260924 15:20:11.175922 11175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bfe967d9462d474b9a4a47d0299042da" candidate_uuid: "6b6f8b2ee66e41d48259916d7f334712" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "766a0be7b2c04fecb0f580f3636fee92"
I20260924 15:20:11.175968 10880 raft_consensus.cc:3037] T bfe967d9462d474b9a4a47d0299042da P e6318bd2eb89456fbf27198154448faf [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:20:11.175984 11175 raft_consensus.cc:3037] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:20:11.176836 11175 raft_consensus.cc:2454] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b6f8b2ee66e41d48259916d7f334712 in term 1.
I20260924 15:20:11.176836 10880 raft_consensus.cc:2454] T bfe967d9462d474b9a4a47d0299042da P e6318bd2eb89456fbf27198154448faf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b6f8b2ee66e41d48259916d7f334712 in term 1.
I20260924 15:20:11.177011 10971 leader_election.cc:302] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b6f8b2ee66e41d48259916d7f334712, e6318bd2eb89456fbf27198154448faf; no voters:
I20260924 15:20:11.177124 11249 raft_consensus.cc:2781] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:20:11.177253 11249 raft_consensus.cc:686] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [term 1 LEADER]: Becoming Leader. State: Replica: 6b6f8b2ee66e41d48259916d7f334712, State: Running, Role: LEADER
I20260924 15:20:11.177340 11249 consensus_queue.cc:237] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766a0be7b2c04fecb0f580f3636fee92" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38831 } } peers { permanent_uuid: "6b6f8b2ee66e41d48259916d7f334712" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 34573 } } peers { permanent_uuid: "e6318bd2eb89456fbf27198154448faf" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34851 } }
I20260924 15:20:11.178079 10732 catalog_manager.cc:5495] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b6f8b2ee66e41d48259916d7f334712 (127.31.12.66). New cstate: current_term: 1 leader_uuid: "6b6f8b2ee66e41d48259916d7f334712" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "766a0be7b2c04fecb0f580f3636fee92" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b6f8b2ee66e41d48259916d7f334712" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 34573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6318bd2eb89456fbf27198154448faf" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34851 } health_report { overall_health: UNKNOWN } } }
I20260924 15:20:11.190727 11240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:11.187683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48406 (local address 127.31.12.66:34573)
0924 15:20:11.187742 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:11.187744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:11.187752 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:20:11.187786 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:11.187787 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:11.187806 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:11.187883 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:20:11.188035 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:11.188713 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:11.189863 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:11.190137 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:11.190195 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:11.190376 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:20:11.190495 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:20:11.190571 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:11.190636 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 15:20:11.190655 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 15:20:11.192684 11175 raft_consensus.cc:1261] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92 [term 1 FOLLOWER]: Refusing update from remote peer 6b6f8b2ee66e41d48259916d7f334712: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:20:11.192685 10880 raft_consensus.cc:1261] T bfe967d9462d474b9a4a47d0299042da P e6318bd2eb89456fbf27198154448faf [term 1 FOLLOWER]: Refusing update from remote peer 6b6f8b2ee66e41d48259916d7f334712: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:20:11.192989 11249 consensus_queue.cc:1035] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [LEADER]: Connected to new peer: Peer: permanent_uuid: "766a0be7b2c04fecb0f580f3636fee92" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:20:11.193094 11253 consensus_queue.cc:1035] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6318bd2eb89456fbf27198154448faf" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:20:11.199540 11258 mvcc.cc:205] Tried to move back new op lower bound from 7332918113042726912 to 7332918112982994944. Current Snapshot: MvccSnapshot[applied={T|T < 7332918113042726912}]
I20260924 15:20:11.199743 11256 mvcc.cc:205] Tried to move back new op lower bound from 7332918113042726912 to 7332918112982994944. Current Snapshot: MvccSnapshot[applied={T|T < 7332918113042726912}]
I20260924 15:20:11.199939 11257 mvcc.cc:205] Tried to move back new op lower bound from 7332918113042726912 to 7332918112982994944. Current Snapshot: MvccSnapshot[applied={T|T < 7332918113042726912}]
I20260924 15:20:11.202668 10731 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:20:11.202756 10731 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:20:11.203743 10731 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 999cf1e1730e408db75b5e83e8cd1118: Sending DeleteTablet for 3 replicas of tablet bfe967d9462d474b9a4a47d0299042da
I20260924 15:20:11.204427 11151 tablet_service.cc:1514] Processing DeleteTablet for tablet bfe967d9462d474b9a4a47d0299042da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:20:11 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:35090
I20260924 15:20:11.204619 11020 tablet_service.cc:1514] Processing DeleteTablet for tablet bfe967d9462d474b9a4a47d0299042da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:20:11 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:48400
I20260924 15:20:11.204805 10879 tablet_service.cc:1514] Processing DeleteTablet for tablet bfe967d9462d474b9a4a47d0299042da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:20:11 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:47600
I20260924 15:20:11.205479 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 10814
I20260924 15:20:11.209736 11266 tablet_replica.cc:330] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712: stopping tablet replica
I20260924 15:20:11.213745 11265 tablet_replica.cc:330] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92: stopping tablet replica
I20260924 15:20:11.213929 11265 raft_consensus.cc:2229] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:20:11.214166 11266 raft_consensus.cc:2229] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:20:11.214341 11265 raft_consensus.cc:2258] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:20:11.214540 11266 raft_consensus.cc:2258] T bfe967d9462d474b9a4a47d0299042da P 6b6f8b2ee66e41d48259916d7f334712 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:20:11.215140 11265 ts_tablet_manager.cc:1905] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 15:20:11.216929 10717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.12.65:34851 (error 108)
I20260924 15:20:11.217270 11265 ts_tablet_manager.cc:1918] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:20:11.217417 11265 log.cc:1196] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/bfe967d9462d474b9a4a47d0299042da
I20260924 15:20:11.218082 11265 ts_tablet_manager.cc:1939] T bfe967d9462d474b9a4a47d0299042da P 766a0be7b2c04fecb0f580f3636fee92: Deleting consensus metadata
I20260924 15:20:11.218557 10716 catalog_manager.cc:4841] TS 766a0be7b2c04fecb0f580f3636fee92 (127.31.12.67:38831): tablet bfe967d9462d474b9a4a47d0299042da (table test-table [id=c2e2a9b38d804bd588d8fb43b3cac7e2]) successfully deleted
I20260924 15:20:11.219036 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 10952
I20260924 15:20:11.221225 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:11.221049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:34851 (local address 127.0.0.1:47616)
0924 15:20:11.221117 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:11.221184 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.65:34851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 15:20:11.221351 10717 catalog_manager.cc:4584] TS e6318bd2eb89456fbf27198154448faf (127.31.12.65:34851): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bfe967d9462d474b9a4a47d0299042da: Network error: Client connection negotiation failed: client connection to 127.31.12.65:34851: connect: Connection refused (error 111)
I20260924 15:20:11.225096 11238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:11.225007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:34573 (local address 127.0.0.1:48408)
0924 15:20:11.225047 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:11.225069 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.66:34573: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 15:20:11.225194 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 11092
I20260924 15:20:11.231140 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 10700
I20260924 15:20:11.238082 31793 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 15:20:11.581931 31793 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 15:20:11.582070 31793 mini_ranger.cc:66] Stopping Ranger...
I20260924 15:20:11.926204 31793 mini_ranger.cc:68] Stopped Ranger
2026-09-24 15:20:11.926 UTC [10264] LOG: received smart shutdown request
2026-09-24 15:20:11.929 UTC [10264] LOG: background worker "logical replication launcher" (PID 10273) exited with exit code 1
2026-09-24 15:20:11.930 UTC [10268] LOG: shutting down
2026-09-24 15:20:12.102 UTC [10264] LOG: database system is shut down
2026-09-24T15:20:12Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48081 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:20:12 dist-test-slave-zwbb krb5kdc[11278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:20:12 dist-test-slave-zwbb krb5kdc[11278](info): set up 2 sockets
Sep 24 15:20:12 dist-test-slave-zwbb krb5kdc[11278](info): commencing operation
krb5kdc: starting...
W20260924 15:20:13.251362 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:20:13 dist-test-slave-zwbb krb5kdc[11278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:20:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:20:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.31.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.31.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.31.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/rangeradmin_127.31.12.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.31.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/rangeradmin_127.31.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.31.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.31.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.31.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/rangerlookup_127.31.12.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.31.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/rangerlookup_127.31.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.31.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/HTTP_127.31.12.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/HTTP_127.31.12.84@KRBTEST.COM.keytab.
I20260924 15:20:13.412273 31793 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskqOs6Q8/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/postgres -l logfile start
2026-09-24 15:20:15.828 UTC [11309] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 15:20:15.828 UTC [11309] LOG: listening on IPv4 address "127.31.12.84", port 42317
2026-09-24 15:20:15.832 UTC [11309] LOG: listening on Unix socket "/tmp/.s.PGSQL.42317"
2026-09-24 15:20:15.867 UTC [11312] LOG: database system was shut down at 2026-09-24 15:20:14 UTC
2026-09-24 15:20:15.871 UTC [11309] LOG: database system is ready to accept connections
I20260924 15:20:16.822680 31793 mini_postgres.cc:96] Postgres bound to 42317
2026-09-24 15:20:16.828 UTC [11321] FATAL: database "slave" does not exist
127.31.12.84:42317 - accepting connections
I20260924 15:20:16.829392 31793 mini_ranger.cc:161] Starting Ranger...
I20260924 15:20:16.844503 31793 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 15:20:17.318148 31793 mini_ranger.cc:87] Created ranger Postgres database
I20260924 15:20:17.318256 31793 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ranger-admin
2026-09-24 15:20:17,580 [I] DB FLAVOR :POSTGRES
2026-09-24 15:20:17,581 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 15:20:17,581 [I] Checking connection..
2026-09-24 15:20:17,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 15:20:17,830 [I] Checking connection passed.
2026-09-24 15:20:17,831 [I] --------- Verifying version history table ---------
2026-09-24 15:20:17,831 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 15:20:18,062 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 15:20:18,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 15:20:18,313 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 15:20:18,313 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 15:20:18,313 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 15:20:18.556 UTC [11387] WARNING: there is no transaction in progress
2026-09-24 15:20:18,563 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 15:20:18,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 15:20:18,823 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 15:20:18,824 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 15:20:18,824 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 15:20:19,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 15:20:19,308 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zwbb.c.gcp-upstream.internal','N') ;"
2026-09-24 15:20:19,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 15:20:19,820 [I] Table x_portal_user does not exist in database ranger
2026-09-24 15:20:19,820 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 15:20:20,072 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 15:20:20,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 15:20:20,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 15:20:20,693 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 15:20:20,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 15:20:21.669 UTC [11499] WARNING: there is no transaction in progress
2026-09-24 15:20:21.685 UTC [11499] WARNING: there is no transaction in progress
2026-09-24 15:20:21.701 UTC [11499] WARNING: there is no transaction in progress
2026-09-24 15:20:21.717 UTC [11499] WARNING: there is no transaction in progress
2026-09-24 15:20:21.732 UTC [11499] WARNING: there is no transaction in progress
2026-09-24 15:20:21.750 UTC [11499] WARNING: there is no transaction in progress
2026-09-24 15:20:21.814 UTC [11499] WARNING: there is no transaction in progress
2026-09-24 15:20:21.821 UTC [11499] WARNING: there is no transaction in progress
2026-09-24 15:20:21.835 UTC [11499] WARNING: there is no transaction in progress
2026-09-24 15:20:21.845 UTC [11499] WARNING: there is no transaction in progress
2026-09-24 15:20:21.857 UTC [11499] WARNING: there is no transaction in progress
2026-09-24 15:20:21.872 UTC [11499] WARNING: there is no transaction in progress
2026-09-24 15:20:21.882 UTC [11499] WARNING: there is no transaction in progress
2026-09-24 15:20:21.888 UTC [11499] WARNING: there is no transaction in progress
2026-09-24 15:20:21.894 UTC [11499] WARNING: there is no transaction in progress
2026-09-24 15:20:21.901 UTC [11499] WARNING: there is no transaction in progress
2026-09-24 15:20:22,516 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 15:20:22,517 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 15:20:22,770 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 15:20:22,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 15:20:23,025 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 15:20:23,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 15:20:23,291 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 15:20:23,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 15:20:23,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 15:20:23,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zwbb.c.gcp-upstream.internal';"
2026-09-24 15:20:24,057 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 15:20:24,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 15:20:24,313 [I] DB_PATCHES have already been applied
I20260924 15:20:24.319453 31793 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ranger-admin:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskqOs6Q8/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 15:20:24.319550 31793 mini_ranger.cc:193] Using host: 127.31.12.84
I20260924 15:20:24.322817 31793 mini_ranger.cc:227] Ranger admin URL: http://127.31.12.84:42259
Sep 24, 2026 3:20:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:20:24 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:20:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 3:20:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 3:20:24 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 3:20:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 3:20:25 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 15:20:25.449143 31793 mini_ranger.cc:160] Time spent starting Ranger: real 8.620s user 0.003s sys 0.005s
Sep 24, 2026 3:20:44 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 3:20:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 3:20:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 3:20:44 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 3:20:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 3:20:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 3:20:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 3:20:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 3:20:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 3:20:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 3:20:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 3:20:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 3:20:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 3:20:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 3:20:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 3:20:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 3:20:44 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 3:20:45 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 15:20:45.868665 31793 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.31.12.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.31.12.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.31.12.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/rangerkms_127.31.12.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.31.12.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/rangerkms_127.31.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.31.12.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.31.12.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.31.12.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/HTTP_127.31.12.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.31.12.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/HTTP_127.31.12.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 15:20:45 dist-test-slave-zwbb krb5kdc[11278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263245, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 15:20:45.949097 31793 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 15:20:45.958796 31793 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 15:20:46.772475 31793 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 15:20:46.772570 31793 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ranger-kms
I20260924 15:20:46.772591 31793 mini_ranger_kms.cc:227] Using postgres at 127.31.12.84:42317
2026-09-24 15:20:46,939 [I] DB FLAVOR :POSTGRES
2026-09-24 15:20:46,939 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 15:20:46,939 [I] Checking connection
2026-09-24 15:20:46,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 15:20:47,199 [I] connection success
2026-09-24 15:20:47,199 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 15:20:47,199 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 15:20:47,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 15:20:47,427 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 15:20:47,428 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 15:20:47,428 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 15:20:47,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.31.12.84:42317/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 15:20:47,674 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 15:20:48.121768 31793 mini_ranger_kms.cc:310] Created kms service
I20260924 15:20:48.292927 31793 mini_ranger_kms.cc:326] Created kudu user
I20260924 15:20:48.337409 31793 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 15:20:48 dist-test-slave-zwbb krb5kdc[11278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263245, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
I20260924 15:20:48.734962 31793 mini_ranger_kms.cc:384] Added ranger policy
I20260924 15:20:48.735193 31793 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ranger-kms:/tmp/dist-test-taskqOs6Q8/build/release/bin/postgresql.jar:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskqOs6Q8/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 15:20:48.735219 31793 mini_ranger_kms.cc:240] Using host: 127.31.12.84
I20260924 15:20:48.738700 31793 mini_ranger_kms.cc:276] Ranger KMS PID: 11680
I20260924 15:20:48.738783 31793 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.31.12.84:58269
Sep 24, 2026 3:20:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 3:20:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 3:20:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 3:20:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 3:20:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 3:20:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 15:20:50 dist-test-slave-zwbb krb5kdc[11278](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790263250, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.12.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:20:50.965175 31793 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.016s user 0.003s sys 0.147s
I20260924 15:20:50.965287 31793 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 15:20:50.965341 31793 mini_ranger_kms.cc:401] 127.31.12.84:58269/kms/v1/keys
Sep 24 15:20:51 dist-test-slave-zwbb krb5kdc[11278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263250, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.12.84@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
Sep 24 15:20:51 dist-test-slave-zwbb krb5kdc[11278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263250, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.31.12.84@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
Sep 24 15:20:53 dist-test-slave-zwbb krb5kdc[11278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:20:53.964474 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:42033
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:42033
--ranger_config_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.12.84:58269/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:20:54.058732 11733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:20:54.058934 11733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:20:54.058956 11733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:20:54.060135 11733 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:20:54.060175 11733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:20:54.060189 11733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:20:54.060202 11733 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:20:54.060214 11733 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:20:54.061777 11733 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/wal
--ranger_kms_url=127.31.12.84:58269/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:42033
--ranger_config_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:42033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11733
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:20:54.062008 11733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:20:54.062245 11733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:20:54.068261 11742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:54.068301 11740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:20:54.068259 11733 server_base.cc:1034] running on GCE node
W20260924 15:20:54.068488 11739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:20:54.068771 11733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:20:54.069065 11733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:20:54.070221 11733 hybrid_clock.cc:648] HybridClock initialized: now 1790263254070198 us; error 39 us; skew 500 ppm
Sep 24 15:20:54 dist-test-slave-zwbb krb5kdc[11278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263254, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:20:54.072733 11733 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:20:54.073202 11733 webserver.cc:460] Webserver started at http://127.31.12.126:39105/ using document root <none> and password file <none>
I20260924 15:20:54.073491 11733 fs_manager.cc:359] Metadata directory not provided
I20260924 15:20:54.073541 11733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:20:54.073669 11733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:20:54 dist-test-slave-zwbb krb5kdc[11278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263254, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
I20260924 15:20:54.125360 11733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "161898fcc0094d18be4a622c754412a4"
format_stamp: "Formatted at 2026-09-24 15:20:54 on dist-test-slave-zwbb"
server_key: "be0368286bb639a52474da70a8a4e92d"
server_key_iv: "9a2ae0a5cb7a4dcaabce700d1690d5af"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:54.125869 11733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "161898fcc0094d18be4a622c754412a4"
format_stamp: "Formatted at 2026-09-24 15:20:54 on dist-test-slave-zwbb"
server_key: "be0368286bb639a52474da70a8a4e92d"
server_key_iv: "9a2ae0a5cb7a4dcaabce700d1690d5af"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:54.127288 11733 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:20:54.139292 11749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:54.139554 11733 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:20:54.139746 11733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "161898fcc0094d18be4a622c754412a4"
format_stamp: "Formatted at 2026-09-24 15:20:54 on dist-test-slave-zwbb"
server_key: "be0368286bb639a52474da70a8a4e92d"
server_key_iv: "9a2ae0a5cb7a4dcaabce700d1690d5af"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:54.139926 11733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:20:54.158048 11733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:20:54.160171 11733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:20:54.160337 11733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:20:54.165344 11733 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:42033
I20260924 15:20:54.165376 11803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:42033 every 8 connection(s)
I20260924 15:20:54.165829 11733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:20:54.166541 11804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:20:54.168689 11804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4: Bootstrap starting.
I20260924 15:20:54.169257 11804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4: Neither blocks nor log segments found. Creating new log.
I20260924 15:20:54.169608 11804 log.cc:824] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4: Log is configured to *not* fsync() on all Append() calls
I20260924 15:20:54.170316 11804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4: No bootstrap required, opened a new log
I20260924 15:20:54.170848 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 11733
I20260924 15:20:54.170966 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:20:54.171151 31793 external_mini_cluster.cc:1428] Setting key 94294202419c138f0e5ef05a828ec307
I20260924 15:20:54.172919 11804 raft_consensus.cc:348] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161898fcc0094d18be4a622c754412a4" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 42033 } }
I20260924 15:20:54.173080 11804 raft_consensus.cc:374] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:20:54.173144 11804 raft_consensus.cc:729] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 161898fcc0094d18be4a622c754412a4, State: Initialized, Role: FOLLOWER
I20260924 15:20:54.173342 11804 consensus_queue.cc:260] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161898fcc0094d18be4a622c754412a4" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 42033 } }
I20260924 15:20:54.173470 11804 raft_consensus.cc:388] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:20:54.173560 11804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:20:54.173621 11804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:20:54.174285 11804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161898fcc0094d18be4a622c754412a4" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 42033 } }
I20260924 15:20:54.174412 11804 leader_election.cc:302] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 161898fcc0094d18be4a622c754412a4; no voters:
I20260924 15:20:54.174620 11804 leader_election.cc:288] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:20:54.174646 11809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:20:54.174875 11809 raft_consensus.cc:686] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [term 1 LEADER]: Becoming Leader. State: Replica: 161898fcc0094d18be4a622c754412a4, State: Running, Role: LEADER
I20260924 15:20:54.174963 11804 sys_catalog.cc:564] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:20:54.175019 11809 consensus_queue.cc:237] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161898fcc0094d18be4a622c754412a4" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 42033 } }
I20260924 15:20:54.175366 11809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "161898fcc0094d18be4a622c754412a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161898fcc0094d18be4a622c754412a4" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 42033 } } }
I20260924 15:20:54.175520 11809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [sys.catalog]: This master's current role is: LEADER
I20260924 15:20:54.175704 11809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 161898fcc0094d18be4a622c754412a4. Latest consensus state: current_term: 1 leader_uuid: "161898fcc0094d18be4a622c754412a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161898fcc0094d18be4a622c754412a4" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 42033 } } }
I20260924 15:20:54.175819 11809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [sys.catalog]: This master's current role is: LEADER
I20260924 15:20:54.175966 11818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:20:54.176595 11818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:20:54.176949 11804 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 15:20:54.177930 11826 mvcc.cc:205] Tried to move back new op lower bound from 7332918289111187456 to 7332918289101385728. Current Snapshot: MvccSnapshot[applied={T|T < 7332918289111187456}]
Sep 24 15:20:54 dist-test-slave-zwbb krb5kdc[11278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:20:54.185593 11818 catalog_manager.cc:1347] Generated new cluster ID: bc8c7bd1dd3e40f18dcd2e240f58162d
I20260924 15:20:54.185734 11818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:20:54.187085 11808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:54.171651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40578 (local address 127.31.12.126:42033)
0924 15:20:54.171858 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:54.171867 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:54.171910 (+ 43us) server_negotiation.cc:407] Connection header received
0924 15:20:54.172934 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:54.172944 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:54.173013 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:54.173158 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:20:54.174069 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:54.175507 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:54.176631 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:54.177221 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:54.179089 (+ 1868us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:54.180238 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:20:54.180242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:20:54.180243 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:20:54.180254 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:20:54.184527 (+ 4273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:54.185064 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:54.185067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:54.185068 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:54.185260 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:54.185547 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:54.185548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:54.185548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:54.186060 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:20:54.186256 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:54.186337 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 15:20:54.186825 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260924 15:20:54.202045 11818 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:20:54.202800 11818 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:20:54.208140 11818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4: Generated new TSK 0
I20260924 15:20:54.208384 11818 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
I20260924 15:20:54.925609 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.12.84:58269/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:42033
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:20:55.015086 11849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:20:55.015323 11849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:20:55.015362 11849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:20:55.016534 11849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:20:55.016606 11849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:20:55.016650 11849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:20:55.018193 11849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.31.12.84:58269/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:42033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:20:55.018453 11849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:20:55.018697 11849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:20:55.023810 11854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:55.023818 11857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:55.024132 11855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:20:55.024240 11849 server_base.cc:1034] running on GCE node
I20260924 15:20:55.024439 11849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:20:55.024714 11849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:20:55.025861 11849 hybrid_clock.cc:648] HybridClock initialized: now 1790263255025832 us; error 44 us; skew 500 ppm
Sep 24 15:20:55 dist-test-slave-zwbb krb5kdc[11278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263255, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:20:55.034144 11849 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:20:55.034688 11849 webserver.cc:460] Webserver started at http://127.31.12.65:35365/ using document root <none> and password file <none>
I20260924 15:20:55.034948 11849 fs_manager.cc:359] Metadata directory not provided
I20260924 15:20:55.035019 11849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:20:55.035172 11849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:20:55 dist-test-slave-zwbb krb5kdc[11278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263255, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
I20260924 15:20:55.048557 11849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "1e26ee780db54b87a1de08dd2890df8a"
format_stamp: "Formatted at 2026-09-24 15:20:55 on dist-test-slave-zwbb"
server_key: "0bffe246f6fab2a35fb6a36b4d544aeb"
server_key_iv: "a37a0deee412818512a2c61f9fb2193a"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:55.048982 11849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "1e26ee780db54b87a1de08dd2890df8a"
format_stamp: "Formatted at 2026-09-24 15:20:55 on dist-test-slave-zwbb"
server_key: "0bffe246f6fab2a35fb6a36b4d544aeb"
server_key_iv: "a37a0deee412818512a2c61f9fb2193a"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:55.056150 11849 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.003s
I20260924 15:20:55.070873 11865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:55.071429 11849 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:20:55.071552 11849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "1e26ee780db54b87a1de08dd2890df8a"
format_stamp: "Formatted at 2026-09-24 15:20:55 on dist-test-slave-zwbb"
server_key: "0bffe246f6fab2a35fb6a36b4d544aeb"
server_key_iv: "a37a0deee412818512a2c61f9fb2193a"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:55.071658 11849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:20:55.105943 11849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:20:55.108201 11849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:20:55.108356 11849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:20:55.108589 11849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:20:55.108899 11849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:20:55.108953 11849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:55.108994 11849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:20:55.109021 11849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:55.115841 11849 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:43637
I20260924 15:20:55.115983 11980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:43637 every 8 connection(s)
I20260924 15:20:55.116287 11849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:20:55 dist-test-slave-zwbb krb5kdc[11278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263255, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:20:55.123303 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 11849
I20260924 15:20:55.123406 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:20:55.123585 31793 external_mini_cluster.cc:1428] Setting key 21d5c86cdcd09889759c8941677e60c1
I20260924 15:20:55.126616 11984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:55.117376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:32903 (local address 127.31.12.126:42033)
0924 15:20:55.117539 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:55.117541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:55.117552 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:20:55.118035 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:55.118036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:55.118050 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:55.118123 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:20:55.118698 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:55.119226 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:55.120225 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:55.120444 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:55.120535 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:55.122778 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:20:55.122780 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:20:55.122780 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:20:55.122788 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:20:55.124991 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:55.125533 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:55.125535 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:55.125535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:55.125605 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:55.125924 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:55.125926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:55.125926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:55.126091 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:20:55.126153 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:55.126431 (+ 278us) server_negotiation.cc:299] Negotiation successful
0924 15:20:55.126520 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":98,"threads_started":1}
I20260924 15:20:55.126806 11983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:55.117134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:42033 (local address 127.31.12.65:32903)
0924 15:20:55.117297 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:55.117337 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:20:55.117903 (+ 566us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:20:55.118196 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:20:55.118200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:20:55.118392 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:20:55.118589 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:55.118594 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:55.119322 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:55.119323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:55.120117 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:55.120119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:55.120539 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:55.120540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:55.120680 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:55.121187 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:20:55.121195 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:20:55.122683 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:20:55.125084 (+ 2401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:20:55.125092 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:20:55.125094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:20:55.125411 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:20:55.125683 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:20:55.125684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:20:55.125684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:20:55.125836 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:20:55.126143 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:20:55.126150 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:20:55.126293 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 15:20:55.126399 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 15:20:55.126553 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260924 15:20:55.127571 11981 heartbeater.cc:347] Connected to a master server at 127.31.12.126:42033
I20260924 15:20:55.127678 11981 heartbeater.cc:464] Registering TS with master...
I20260924 15:20:55.127866 11981 heartbeater.cc:511] Master 127.31.12.126:42033 requested a full tablet report, sending...
I20260924 15:20:55.128430 11767 ts_manager.cc:194] Registered new tserver with Master: 1e26ee780db54b87a1de08dd2890df8a (127.31.12.65:43637)
I20260924 15:20:55.129285 11767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:32903
WARNING: no policy specified for kudu/127.31.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
I20260924 15:20:55.154481 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.12.84:58269/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:42033
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:20:55.267076 11987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:20:55.267349 11987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:20:55.267388 11987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:20:55.269013 11987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:20:55.269074 11987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:20:55.269112 11987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:20:55.271268 11987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.31.12.84:58269/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:42033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:20:55.271517 11987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:20:55.271842 11987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:20:55.278261 11998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:55.278366 11995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:55.278266 11994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:20:55.279197 11987 server_base.cc:1034] running on GCE node
I20260924 15:20:55.279412 11987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:20:55.279698 11987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:20:55.280875 11987 hybrid_clock.cc:648] HybridClock initialized: now 1790263255280853 us; error 36 us; skew 500 ppm
Sep 24 15:20:55 dist-test-slave-zwbb krb5kdc[11278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263255, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:20:55.284118 11987 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:20:55.284693 11987 webserver.cc:460] Webserver started at http://127.31.12.66:43155/ using document root <none> and password file <none>
I20260924 15:20:55.284966 11987 fs_manager.cc:359] Metadata directory not provided
I20260924 15:20:55.285038 11987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:20:55.285234 11987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:20:55 dist-test-slave-zwbb krb5kdc[11278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263255, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
Sep 24 15:20:55 dist-test-slave-zwbb krb5kdc[11278](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790263255, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:20:55.300375 11987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "dfc620b728c84aef988d21fe80f201d3"
format_stamp: "Formatted at 2026-09-24 15:20:55 on dist-test-slave-zwbb"
server_key: "6cccf556e5d50da6fb50a58335e3a524"
server_key_iv: "58cd803beff380b3f32c1550f4c797f0"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:55.300838 11987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "dfc620b728c84aef988d21fe80f201d3"
format_stamp: "Formatted at 2026-09-24 15:20:55 on dist-test-slave-zwbb"
server_key: "6cccf556e5d50da6fb50a58335e3a524"
server_key_iv: "58cd803beff380b3f32c1550f4c797f0"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:55.305896 11987 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 15:20:55.334987 12005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:55.335212 11987 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.001s sys 0.000s
I20260924 15:20:55.335326 11987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "dfc620b728c84aef988d21fe80f201d3"
format_stamp: "Formatted at 2026-09-24 15:20:55 on dist-test-slave-zwbb"
server_key: "6cccf556e5d50da6fb50a58335e3a524"
server_key_iv: "58cd803beff380b3f32c1550f4c797f0"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:55.335443 11987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:20:55.354748 11987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:20:55.357055 11987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:20:55.357249 11987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:20:55.357532 11987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:20:55.357883 11987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:20:55.357928 11987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:55.357957 11987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:20:55.358021 11987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:55.365561 11987 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:35647
I20260924 15:20:55.365638 12119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:35647 every 8 connection(s)
I20260924 15:20:55.365933 11987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:20:55.370777 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 11987
I20260924 15:20:55.370867 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:20:55.371063 31793 external_mini_cluster.cc:1428] Setting key 46e6df7ccfff278cd17a8fa91fc98f0e
Sep 24 15:20:55 dist-test-slave-zwbb krb5kdc[11278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263255, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:20:55.376067 11984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:55.366659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:52091 (local address 127.31.12.126:42033)
0924 15:20:55.366708 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:55.366711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:55.366931 (+ 220us) server_negotiation.cc:407] Connection header received
0924 15:20:55.367492 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:55.367493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:55.367507 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:55.367551 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:20:55.368112 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:55.368615 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:55.369681 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:55.369907 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:55.369994 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:55.372489 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:20:55.372491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:20:55.372492 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:20:55.372499 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:20:55.374300 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:55.374809 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:55.374811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:55.374811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:55.374875 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:55.375238 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:55.375239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:55.375239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:55.375381 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:20:55.375434 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:55.375812 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 15:20:55.375914 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:20:55.376644 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:55.366660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:42033 (local address 127.31.12.66:52091)
0924 15:20:55.366817 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:55.366857 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:20:55.367381 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:20:55.367642 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:20:55.367646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:20:55.367823 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:20:55.368007 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:55.368011 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:55.368723 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:55.368724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:55.369591 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:55.369594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:55.370023 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:55.370024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:55.370141 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:55.370680 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:20:55.370691 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:20:55.372380 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:20:55.374385 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:20:55.374392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:20:55.374394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:20:55.374715 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:20:55.374947 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:20:55.374948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:20:55.374948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:20:55.375160 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:20:55.375485 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:20:55.375492 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:20:55.375660 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 15:20:55.376220 (+ 560us) client_negotiation.cc:240] Negotiation successful
0924 15:20:55.376383 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260924 15:20:55.377192 12120 heartbeater.cc:347] Connected to a master server at 127.31.12.126:42033
I20260924 15:20:55.377326 12120 heartbeater.cc:464] Registering TS with master...
I20260924 15:20:55.377578 12120 heartbeater.cc:511] Master 127.31.12.126:42033 requested a full tablet report, sending...
I20260924 15:20:55.378008 11767 ts_manager.cc:194] Registered new tserver with Master: dfc620b728c84aef988d21fe80f201d3 (127.31.12.66:35647)
I20260924 15:20:55.378373 11767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:52091
WARNING: no policy specified for kudu/127.31.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
I20260924 15:20:55.400833 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.12.84:58269/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:42033
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:20:55.516130 12125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:20:55.516395 12125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:20:55.516429 12125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:20:55.518070 12125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:20:55.518186 12125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:20:55.518266 12125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:20:55.520350 12125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.31.12.84:58269/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:42033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:20:55.520574 12125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:20:55.520851 12125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:20:55.527295 12134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:55.527432 12132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:55.527701 12131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:20:55.527830 12125 server_base.cc:1034] running on GCE node
I20260924 15:20:55.528095 12125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:20:55.528340 12125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:20:55.529506 12125 hybrid_clock.cc:648] HybridClock initialized: now 1790263255529473 us; error 44 us; skew 500 ppm
Sep 24 15:20:55 dist-test-slave-zwbb krb5kdc[11278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263255, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:20:55.532498 12125 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:20:55.532904 12125 webserver.cc:460] Webserver started at http://127.31.12.67:43257/ using document root <none> and password file <none>
I20260924 15:20:55.533126 12125 fs_manager.cc:359] Metadata directory not provided
I20260924 15:20:55.533195 12125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:20:55.533349 12125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 15:20:55 dist-test-slave-zwbb krb5kdc[11278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263255, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
I20260924 15:20:55.549345 12125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "ffac70834cbf4df7aa30a3acfcef060b"
format_stamp: "Formatted at 2026-09-24 15:20:55 on dist-test-slave-zwbb"
server_key: "479706a68591e17fab09c3c8c1699853"
server_key_iv: "8b1a91c6681720d8a13274b69e538ca5"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:55.549839 12125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "ffac70834cbf4df7aa30a3acfcef060b"
format_stamp: "Formatted at 2026-09-24 15:20:55 on dist-test-slave-zwbb"
server_key: "479706a68591e17fab09c3c8c1699853"
server_key_iv: "8b1a91c6681720d8a13274b69e538ca5"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:55.551369 12125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:20:55.572795 12141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:55.573069 12125 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260924 15:20:55.573201 12125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "ffac70834cbf4df7aa30a3acfcef060b"
format_stamp: "Formatted at 2026-09-24 15:20:55 on dist-test-slave-zwbb"
server_key: "479706a68591e17fab09c3c8c1699853"
server_key_iv: "8b1a91c6681720d8a13274b69e538ca5"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:55.573309 12125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:20:55.595273 12125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:20:55.597837 12125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:20:55.598002 12125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:20:55.598272 12125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:20:55.598636 12125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:20:55.598683 12125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:55.598721 12125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:20:55.598743 12125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:55.605885 12125 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:33055
I20260924 15:20:55.606271 12125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:20:55.607270 12254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:33055 every 8 connection(s)
I20260924 15:20:55.608131 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 12125
I20260924 15:20:55.608224 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:20:55.608438 31793 external_mini_cluster.cc:1428] Setting key 6dbd2c8cafbbcb558123e9e2eb43b279
Sep 24 15:20:55 dist-test-slave-zwbb krb5kdc[11278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263255, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:20:55.617214 11984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:55.607113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:34939 (local address 127.31.12.126:42033)
0924 15:20:55.607162 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:55.607164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:55.607590 (+ 426us) server_negotiation.cc:407] Connection header received
0924 15:20:55.608429 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:55.608431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:55.608444 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:55.608477 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:20:55.609690 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:55.610196 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:55.611379 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:55.611588 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:55.611672 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:55.613938 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:20:55.613940 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:20:55.613943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:20:55.613950 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:20:55.615614 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:55.616117 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:55.616118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:55.616118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:55.616178 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:55.616450 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:55.616451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:55.616451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:55.616582 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:20:55.616627 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:55.617009 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 15:20:55.617106 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:20:55.617903 12257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:55.607302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:42033 (local address 127.31.12.67:34939)
0924 15:20:55.607450 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:55.607491 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:20:55.608293 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:20:55.608682 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:20:55.608687 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:20:55.608931 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:20:55.609617 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:55.609624 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:55.610294 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:55.610295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:55.611276 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:55.611279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:55.611695 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:55.611697 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:55.611813 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:55.612333 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:20:55.612341 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:20:55.613841 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:20:55.615712 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:20:55.615719 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:20:55.615721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:20:55.616009 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:20:55.616250 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:20:55.616250 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:20:55.616251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:20:55.616380 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:20:55.616674 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:20:55.616680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:20:55.616837 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 15:20:55.617511 (+ 674us) client_negotiation.cc:240] Negotiation successful
0924 15:20:55.617677 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":91,"threads_started":1}
I20260924 15:20:55.618438 12255 heartbeater.cc:347] Connected to a master server at 127.31.12.126:42033
I20260924 15:20:55.618566 12255 heartbeater.cc:464] Registering TS with master...
I20260924 15:20:55.618749 12255 heartbeater.cc:511] Master 127.31.12.126:42033 requested a full tablet report, sending...
I20260924 15:20:55.619145 11767 ts_manager.cc:194] Registered new tserver with Master: ffac70834cbf4df7aa30a3acfcef060b (127.31.12.67:33055)
I20260924 15:20:55.619606 11767 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:34939
I20260924 15:20:55.620693 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:20:55.621376 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 11849
I20260924 15:20:55.627292 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 11987
I20260924 15:20:55.633742 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 12125
I20260924 15:20:55.640674 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 11733
I20260924 15:20:55.645872 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:42033
--webserver_interface=127.31.12.126
--webserver_port=39105
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:42033
--ranger_config_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.12.84:58269/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.12.84:58269/kms with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:20:55.725589 12258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:20:55.725785 12258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:20:55.725806 12258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:20:55.726929 12258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:20:55.726971 12258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:20:55.726986 12258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:20:55.726998 12258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:20:55.727010 12258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:20:55.728593 12258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.12.84:58269/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:42033
--ranger_config_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:42033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=39105
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:20:55.728824 12258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:20:55.729056 12258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:20:55.733796 12265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:55.733754 12264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:20:55.733883 12258 server_base.cc:1034] running on GCE node
W20260924 15:20:55.733753 12267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:20:55.734319 12258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:20:55.734618 12258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:20:55.735790 12258 hybrid_clock.cc:648] HybridClock initialized: now 1790263255735777 us; error 28 us; skew 500 ppm
Sep 24 15:20:55 dist-test-slave-zwbb krb5kdc[11278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263255, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:20:55.738191 12258 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:20:55.738698 12258 webserver.cc:460] Webserver started at http://127.31.12.126:39105/ using document root <none> and password file <none>
I20260924 15:20:55.738916 12258 fs_manager.cc:359] Metadata directory not provided
I20260924 15:20:55.738996 12258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 15:20:55 dist-test-slave-zwbb krb5kdc[11278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263255, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
I20260924 15:20:55.776994 12258 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:20:55.777858 12275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:55.778059 12258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:20:55.778172 12258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "161898fcc0094d18be4a622c754412a4"
format_stamp: "Formatted at 2026-09-24 15:20:54 on dist-test-slave-zwbb"
server_key: "be0368286bb639a52474da70a8a4e92d"
server_key_iv: "9a2ae0a5cb7a4dcaabce700d1690d5af"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:55.778546 12258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:20:55.792851 12258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:20:55.794943 12258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:20:55.795117 12258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:20:55.799186 12258 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:42033
I20260924 15:20:55.799223 12327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:42033 every 8 connection(s)
I20260924 15:20:55.799559 12258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:20:55.800719 12328 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:20:55.801172 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 12258
I20260924 15:20:55.801431 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.12.84:58269/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:43637
--local_ip_for_outbound_sockets=127.31.12.65
--tserver_master_addrs=127.31.12.126:42033
--webserver_port=35365
--webserver_interface=127.31.12.65
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.12.84:58269/kms with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:20:55.801473 12328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4: Bootstrap starting.
I20260924 15:20:55.809827 12328 log.cc:824] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4: Log is configured to *not* fsync() on all Append() calls
I20260924 15:20:55.811544 12328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:20:55.811836 12328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4: Bootstrap complete.
I20260924 15:20:55.814276 12328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161898fcc0094d18be4a622c754412a4" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 42033 } }
I20260924 15:20:55.814532 12328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 161898fcc0094d18be4a622c754412a4, State: Initialized, Role: FOLLOWER
I20260924 15:20:55.814734 12328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161898fcc0094d18be4a622c754412a4" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 42033 } }
I20260924 15:20:55.814831 12328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:20:55.814888 12328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:20:55.814944 12328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [term 1 FOLLOWER]: Advancing to term 2
I20260924 15:20:55.815886 12328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161898fcc0094d18be4a622c754412a4" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 42033 } }
I20260924 15:20:55.816013 12328 leader_election.cc:302] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 161898fcc0094d18be4a622c754412a4; no voters:
I20260924 15:20:55.816217 12328 leader_election.cc:288] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 15:20:55.816263 12332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 15:20:55.816362 12332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [term 2 LEADER]: Becoming Leader. State: Replica: 161898fcc0094d18be4a622c754412a4, State: Running, Role: LEADER
I20260924 15:20:55.816452 12332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161898fcc0094d18be4a622c754412a4" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 42033 } }
I20260924 15:20:55.816481 12328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:20:55.816797 12334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 161898fcc0094d18be4a622c754412a4. Latest consensus state: current_term: 2 leader_uuid: "161898fcc0094d18be4a622c754412a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161898fcc0094d18be4a622c754412a4" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 42033 } } }
I20260924 15:20:55.816871 12334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [sys.catalog]: This master's current role is: LEADER
I20260924 15:20:55.817056 12333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "161898fcc0094d18be4a622c754412a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "161898fcc0094d18be4a622c754412a4" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 42033 } } }
I20260924 15:20:55.817157 12333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4 [sys.catalog]: This master's current role is: LEADER
I20260924 15:20:55.817170 12336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:20:55.817696 12336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:20:55.817837 12336 catalog_manager.cc:1259] Loaded cluster ID: bc8c7bd1dd3e40f18dcd2e240f58162d
I20260924 15:20:55.817878 12336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:20:55.818463 12328 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 15:20:55.821171 12336 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:20:55.821907 12336 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4: Loaded TSK: 0
I20260924 15:20:55.822269 12336 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 15:20:55.910212 12330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:20:55.910442 12330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:20:55.910480 12330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:20:55.911613 12330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:20:55.911680 12330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:20:55.911725 12330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:20:55.913142 12330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.12.84:58269/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:43637
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=35365
--tserver_master_addrs=127.31.12.126:42033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:20:55.913404 12330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:20:55.913638 12330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:20:55.919000 12366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:20:55.919314 12330 server_base.cc:1034] running on GCE node
W20260924 15:20:55.919500 12369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:55.923381 12367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:20:55.923669 12330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:20:55.923942 12330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:20:55.925097 12330 hybrid_clock.cc:648] HybridClock initialized: now 1790263255925079 us; error 41 us; skew 500 ppm
Sep 24 15:20:55 dist-test-slave-zwbb krb5kdc[11278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263255, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:20:55.928000 12330 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:20:55.928427 12330 webserver.cc:460] Webserver started at http://127.31.12.65:35365/ using document root <none> and password file <none>
I20260924 15:20:55.928617 12330 fs_manager.cc:359] Metadata directory not provided
I20260924 15:20:55.928663 12330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 15:20:55 dist-test-slave-zwbb krb5kdc[11278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263255, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
I20260924 15:20:55.942482 12330 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 15:20:55.943841 12377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:55.944023 12330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:20:55.944119 12330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "1e26ee780db54b87a1de08dd2890df8a"
format_stamp: "Formatted at 2026-09-24 15:20:55 on dist-test-slave-zwbb"
server_key: "0bffe246f6fab2a35fb6a36b4d544aeb"
server_key_iv: "a37a0deee412818512a2c61f9fb2193a"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:55.944607 12330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:20:55.959426 12330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:20:55.961620 12330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:20:55.961764 12330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:20:55.961987 12330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:20:55.962312 12330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:20:55.962361 12330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:55.962389 12330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:20:55.962423 12330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:55.968513 12330 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:43637
I20260924 15:20:55.968999 12330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:20:55.974218 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 12330
I20260924 15:20:55.974514 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.12.84:58269/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:35647
--local_ip_for_outbound_sockets=127.31.12.66
--tserver_master_addrs=127.31.12.126:42033
--webserver_port=43155
--webserver_interface=127.31.12.66
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.12.84:58269/kms with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:20:55.977880 12493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:43637 every 8 connection(s)
Sep 24 15:20:55 dist-test-slave-zwbb krb5kdc[11278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263255, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:20:55.990576 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:55.978642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:57073 (local address 127.31.12.126:42033)
0924 15:20:55.978922 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:55.978934 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:55.979829 (+ 895us) server_negotiation.cc:407] Connection header received
0924 15:20:55.979977 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:55.979982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:55.980038 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:55.980155 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:20:55.980917 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:55.982296 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:55.983758 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:55.984335 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:55.984397 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:55.986720 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:20:55.986727 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:20:55.986727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:20:55.986742 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:20:55.988598 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:55.989123 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:55.989127 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:55.989128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:55.989246 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:55.989561 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:55.989562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:55.989562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:55.989818 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:20:55.989915 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:55.990236 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 15:20:55.990346 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":92,"threads_started":1}
I20260924 15:20:55.991228 12497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:55.978561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:42033 (local address 127.31.12.65:57073)
0924 15:20:55.979018 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:55.979064 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 15:20:55.979673 (+ 609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:20:55.980300 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:20:55.980303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:20:55.980538 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:20:55.980792 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:55.980798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:55.982427 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:55.982428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:55.983653 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:55.983656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:55.984520 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:55.984521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:55.984611 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:55.985207 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:20:55.985215 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:20:55.986602 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:20:55.988701 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:20:55.988708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:20:55.988710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:20:55.989015 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:20:55.989331 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:20:55.989331 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:20:55.989332 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:20:55.989483 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:20:55.989903 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:20:55.989909 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:20:55.990055 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 15:20:55.990162 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 15:20:55.990340 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":97,"threads_started":1}
I20260924 15:20:55.992092 12494 heartbeater.cc:347] Connected to a master server at 127.31.12.126:42033
I20260924 15:20:55.992189 12494 heartbeater.cc:464] Registering TS with master...
I20260924 15:20:55.992388 12494 heartbeater.cc:511] Master 127.31.12.126:42033 requested a full tablet report, sending...
W20260924 15:20:55.993271 12494 heartbeater.cc:644] Failed to heartbeat to 127.31.12.126:42033 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 15:20:55.993358 12494 heartbeater.cc:464] Registering TS with master...
I20260924 15:20:55.993419 12494 heartbeater.cc:511] Master 127.31.12.126:42033 requested a full tablet report, sending...
I20260924 15:20:55.993733 12494 heartbeater.cc:464] Registering TS with master...
I20260924 15:20:55.993808 12494 heartbeater.cc:511] Master 127.31.12.126:42033 requested a full tablet report, sending...
W20260924 15:20:55.994076 12494 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 15:20:56.096959 12496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:20:56.097231 12496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:20:56.097266 12496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:20:56.098912 12496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:20:56.098990 12496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:20:56.099033 12496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:20:56.101003 12496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.12.84:58269/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:35647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=43155
--tserver_master_addrs=127.31.12.126:42033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:20:56.101264 12496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:20:56.101559 12496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:20:56.107316 12504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:56.107316 12503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:56.107868 12506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:20:56.108494 12496 server_base.cc:1034] running on GCE node
I20260924 15:20:56.108702 12496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:20:56.108971 12496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:20:56.110131 12496 hybrid_clock.cc:648] HybridClock initialized: now 1790263256110116 us; error 30 us; skew 500 ppm
Sep 24 15:20:56 dist-test-slave-zwbb krb5kdc[11278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263256, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:20:56.112962 12496 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:20:56.113459 12496 webserver.cc:460] Webserver started at http://127.31.12.66:43155/ using document root <none> and password file <none>
I20260924 15:20:56.113773 12496 fs_manager.cc:359] Metadata directory not provided
I20260924 15:20:56.113842 12496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 15:20:56 dist-test-slave-zwbb krb5kdc[11278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263256, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
I20260924 15:20:56.128515 12496 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:20:56.129603 12515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:56.129802 12496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:20:56.129884 12496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "dfc620b728c84aef988d21fe80f201d3"
format_stamp: "Formatted at 2026-09-24 15:20:55 on dist-test-slave-zwbb"
server_key: "6cccf556e5d50da6fb50a58335e3a524"
server_key_iv: "58cd803beff380b3f32c1550f4c797f0"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:56.130366 12496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:20:56.147018 12496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:20:56.149238 12496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:20:56.149398 12496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:20:56.149729 12496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:20:56.150125 12496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:20:56.150177 12496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:56.150207 12496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:20:56.150225 12496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:56.157608 12496 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:35647
I20260924 15:20:56.157763 12633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:35647 every 8 connection(s)
I20260924 15:20:56.158084 12496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:20:56 dist-test-slave-zwbb krb5kdc[11278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263256, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:20:56.164747 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 12496
I20260924 15:20:56.165076 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.31.12.84:58269/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:33055
--local_ip_for_outbound_sockets=127.31.12.67
--tserver_master_addrs=127.31.12.126:42033
--webserver_port=43257
--webserver_interface=127.31.12.67
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.31.12.84:58269/kms with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:20:56.169191 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:56.158901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:51621 (local address 127.31.12.126:42033)
0924 15:20:56.158951 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:56.158953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:56.159200 (+ 247us) server_negotiation.cc:407] Connection header received
0924 15:20:56.159759 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:56.159761 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:56.159776 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:56.159861 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:20:56.160451 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:56.161261 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:56.162391 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:56.162669 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:56.162755 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:56.165384 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:20:56.165386 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:20:56.165386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:20:56.165394 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:20:56.167319 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:56.167737 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:56.167739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:56.167739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:56.167804 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:56.168373 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:56.168374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:56.168375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:56.168533 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:20:56.168601 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:56.168994 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 15:20:56.169094 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:20:56.169301 12636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:56.158906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:42033 (local address 127.31.12.66:51621)
0924 15:20:56.159069 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:56.159110 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:20:56.159583 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:20:56.159940 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:20:56.159944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:20:56.160142 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:20:56.160346 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:56.160351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:56.161368 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:56.161369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:56.162298 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:56.162301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:56.162784 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:56.162786 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:56.162907 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:56.163460 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:20:56.163468 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:20:56.164998 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:20:56.167410 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:20:56.167418 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:20:56.167420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:20:56.167654 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:20:56.168117 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:20:56.168118 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:20:56.168119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:20:56.168284 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:20:56.168630 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:20:56.168637 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:20:56.168818 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 15:20:56.168936 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 15:20:56.169100 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":89,"threads_started":1}
I20260924 15:20:56.169773 12634 heartbeater.cc:347] Connected to a master server at 127.31.12.126:42033
I20260924 15:20:56.169871 12634 heartbeater.cc:464] Registering TS with master...
I20260924 15:20:56.170073 12634 heartbeater.cc:511] Master 127.31.12.126:42033 requested a full tablet report, sending...
W20260924 15:20:56.170811 12634 heartbeater.cc:644] Failed to heartbeat to 127.31.12.126:42033 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 15:20:56.170892 12634 heartbeater.cc:464] Registering TS with master...
I20260924 15:20:56.170967 12634 heartbeater.cc:511] Master 127.31.12.126:42033 requested a full tablet report, sending...
I20260924 15:20:56.171392 12634 heartbeater.cc:464] Registering TS with master...
I20260924 15:20:56.171476 12634 heartbeater.cc:511] Master 127.31.12.126:42033 requested a full tablet report, sending...
W20260924 15:20:56.171813 12634 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 15:20:56.259754 12637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:20:56.259994 12637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:20:56.260031 12637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:20:56.261767 12637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:20:56.261852 12637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:20:56.261897 12637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:20:56.263556 12637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.31.12.84:58269/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:33055
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=43257
--tserver_master_addrs=127.31.12.126:42033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:20:56.263839 12637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:20:56.264068 12637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:20:56.270529 12643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:56.270802 12645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:56.270972 12642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:20:56.271353 12637 server_base.cc:1034] running on GCE node
I20260924 15:20:56.271626 12637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:20:56.271886 12637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:20:56.273072 12637 hybrid_clock.cc:648] HybridClock initialized: now 1790263256273048 us; error 35 us; skew 500 ppm
Sep 24 15:20:56 dist-test-slave-zwbb krb5kdc[11278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263256, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:20:56.276232 12637 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:20:56.276754 12637 webserver.cc:460] Webserver started at http://127.31.12.67:43257/ using document root <none> and password file <none>
I20260924 15:20:56.277016 12637 fs_manager.cc:359] Metadata directory not provided
I20260924 15:20:56.277087 12637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 15:20:56 dist-test-slave-zwbb krb5kdc[11278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263256, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
I20260924 15:20:56.292330 12637 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.001s
I20260924 15:20:56.293315 12653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:56.293551 12637 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:20:56.293666 12637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "ffac70834cbf4df7aa30a3acfcef060b"
format_stamp: "Formatted at 2026-09-24 15:20:55 on dist-test-slave-zwbb"
server_key: "479706a68591e17fab09c3c8c1699853"
server_key_iv: "8b1a91c6681720d8a13274b69e538ca5"
server_key_version: "kuduclusterkey@0"
I20260924 15:20:56.294159 12637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:20:56.346058 12637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:20:56.348632 12637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:20:56.348790 12637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:20:56.349062 12637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:20:56.349478 12637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:20:56.349525 12637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:56.349562 12637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:20:56.349581 12637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:56.357172 12637 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:33055
I20260924 15:20:56.357702 12637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:20:56.358260 12768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:33055 every 8 connection(s)
I20260924 15:20:56.364127 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 12637
Sep 24 15:20:56 dist-test-slave-zwbb krb5kdc[11278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263256, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:20:56.368369 12498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:56.358785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:55119 (local address 127.31.12.126:42033)
0924 15:20:56.358831 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:56.358833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:56.359042 (+ 209us) server_negotiation.cc:407] Connection header received
0924 15:20:56.359749 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:56.359750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:56.359762 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:56.359794 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:20:56.360589 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:56.361333 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:56.362454 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:56.362667 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:56.362756 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:56.365494 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:20:56.365495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:20:56.365495 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:20:56.365502 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:20:56.366959 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:56.367377 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:56.367378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:56.367379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:56.367441 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:56.367717 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:56.367718 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:56.367719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:56.367837 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:20:56.367893 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:56.368197 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 15:20:56.368281 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:20:56.368544 12771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:56.358444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:42033 (local address 127.31.12.67:55119)
0924 15:20:56.358916 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:56.358962 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 15:20:56.359597 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:20:56.359972 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:20:56.359976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:20:56.360201 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:20:56.360469 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:56.360475 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:56.361480 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:56.361481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:56.362362 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:56.362364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:56.362785 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:56.362787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:56.362909 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:56.363488 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:20:56.363497 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:20:56.365376 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:20:56.367069 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:20:56.367076 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:20:56.367078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:20:56.367298 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:20:56.367509 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:20:56.367510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:20:56.367511 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:20:56.367647 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:20:56.367920 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:20:56.367927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:20:56.368055 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 15:20:56.368164 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 15:20:56.368303 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":95,"threads_started":1}
I20260924 15:20:56.369010 12769 heartbeater.cc:347] Connected to a master server at 127.31.12.126:42033
I20260924 15:20:56.369115 12769 heartbeater.cc:464] Registering TS with master...
I20260924 15:20:56.369289 12769 heartbeater.cc:511] Master 127.31.12.126:42033 requested a full tablet report, sending...
W20260924 15:20:56.370088 12769 heartbeater.cc:644] Failed to heartbeat to 127.31.12.126:42033 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 15:20:56.370172 12769 heartbeater.cc:464] Registering TS with master...
I20260924 15:20:56.370500 12769 heartbeater.cc:511] Master 127.31.12.126:42033 requested a full tablet report, sending...
I20260924 15:20:56.370963 12769 heartbeater.cc:464] Registering TS with master...
I20260924 15:20:56.371057 12769 heartbeater.cc:511] Master 127.31.12.126:42033 requested a full tablet report, sending...
W20260924 15:20:56.371354 12769 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 15:20:56.372212 12774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:56.364728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40592 (local address 127.31.12.126:42033)
0924 15:20:56.364841 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:56.364857 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:56.364867 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:20:56.364887 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:56.364888 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:56.364899 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:56.364976 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:20:56.365770 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:56.366376 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:56.367528 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:56.367823 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:56.367884 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:56.369349 (+ 1465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:20:56.369398 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:20:56.369398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:20:56.369405 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:20:56.371120 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:56.371484 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:56.371485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:56.371485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:56.371556 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:56.371769 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:56.371769 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:56.371770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:56.371862 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:20:56.371967 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:56.372083 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 15:20:56.372145 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
Sep 24 15:20:56 dist-test-slave-zwbb krb5kdc[11278](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790263256, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:20:56.994284 12494 heartbeater.cc:464] Registering TS with master...
I20260924 15:20:56.994437 12494 heartbeater.cc:511] Master 127.31.12.126:42033 requested a full tablet report, sending...
I20260924 15:20:56.994890 12292 ts_manager.cc:194] Registered new tserver with Master: 1e26ee780db54b87a1de08dd2890df8a (127.31.12.65:43637)
I20260924 15:20:56.995882 12292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:57073
I20260924 15:20:57.172009 12634 heartbeater.cc:464] Registering TS with master...
I20260924 15:20:57.172176 12634 heartbeater.cc:511] Master 127.31.12.126:42033 requested a full tablet report, sending...
I20260924 15:20:57.172514 12292 ts_manager.cc:194] Registered new tserver with Master: dfc620b728c84aef988d21fe80f201d3 (127.31.12.66:35647)
I20260924 15:20:57.173058 12292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:51621
Sep 24 15:20:57 dist-test-slave-zwbb krb5kdc[11278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263256, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
I20260924 15:20:57.371547 12769 heartbeater.cc:464] Registering TS with master...
I20260924 15:20:57.371714 12769 heartbeater.cc:511] Master 127.31.12.126:42033 requested a full tablet report, sending...
I20260924 15:20:57.372004 12292 ts_manager.cc:194] Registered new tserver with Master: ffac70834cbf4df7aa30a3acfcef060b (127.31.12.67:33055)
I20260924 15:20:57.372532 12292 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:55119
I20260924 15:20:57.381505 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:20:57.390899 12791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:57.382632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.31.12.126:42033)
0924 15:20:57.382909 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:57.382911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:57.383068 (+ 157us) server_negotiation.cc:407] Connection header received
0924 15:20:57.383091 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:57.383091 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:57.383104 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:57.383156 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:20:57.383858 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.384651 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.385725 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.386036 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.386095 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:57.387616 (+ 1521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:20:57.387617 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:20:57.387617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:20:57.387625 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:20:57.389395 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:57.389819 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:57.389820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:57.389821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:57.389877 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:57.390057 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:57.390058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:57.390058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:57.390194 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:20:57.390283 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:57.390716 (+ 433us) server_negotiation.cc:299] Negotiation successful
0924 15:20:57.390795 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":68,"threads_started":1}
I20260924 15:20:57.393256 12292 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:20:57.395509 12292 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:20:57.411229 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:57.405303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:35647 (local address 127.0.0.1:41944)
0924 15:20:57.406310 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:57.406318 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:20:57.406351 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:20:57.406622 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:20:57.406625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:20:57.406627 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:20:57.406699 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:57.406701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.408267 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.408268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:57.409584 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:57.409588 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.410924 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.410925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:57.411008 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:57.411021 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 15:20:57.411086 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 15:20:57.411137 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":998,"thread_start_us":59,"threads_started":1}
I20260924 15:20:57.411680 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:57.405506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.31.12.66:35647)
0924 15:20:57.405932 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:57.405942 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:57.406411 (+ 469us) server_negotiation.cc:407] Connection header received
0924 15:20:57.406449 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:57.406451 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:57.406468 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:57.406533 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:20:57.406771 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.408183 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.409679 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.410748 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.410826 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:57.410887 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:57.411561 (+ 674us) server_negotiation.cc:299] Negotiation successful
0924 15:20:57.411606 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":55,"threads_started":1}
I20260924 15:20:57.412621 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 158caa1bae9f431abe24af89f334e557 (DEFAULT_TABLE table=test-table [id=709c339239bf4554ae8f28113f942085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:20:57.412986 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 158caa1bae9f431abe24af89f334e557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:20:57.416500 12803 tablet_bootstrap.cc:492] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3: Bootstrap starting.
I20260924 15:20:57.417177 12803 tablet_bootstrap.cc:654] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3: Neither blocks nor log segments found. Creating new log.
I20260924 15:20:57.417580 12803 log.cc:824] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3: Log is configured to *not* fsync() on all Append() calls
I20260924 15:20:57.421120 12797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:57.405122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:33055 (local address 127.0.0.1:35178)
0924 15:20:57.406210 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:57.406218 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:20:57.406249 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:20:57.417715 (+ 11466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:20:57.417716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:20:57.417719 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:20:57.417767 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:57.417769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.418665 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.418666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:57.419835 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:57.419838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.420883 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.420884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:57.420946 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:57.420950 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:20:57.420998 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 15:20:57.421036 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1076,"thread_start_us":68,"threads_started":1}
I20260924 15:20:57.421638 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:57.405214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35178 (local address 127.31.12.67:33055)
0924 15:20:57.417427 (+ 12213us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:57.417433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:57.417473 (+ 40us) server_negotiation.cc:407] Connection header received
0924 15:20:57.417511 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:57.417513 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:57.417531 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:57.417621 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:20:57.417854 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.418554 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.419947 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.420722 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.420785 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:57.420840 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:57.421498 (+ 658us) server_negotiation.cc:299] Negotiation successful
0924 15:20:57.421539 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12165,"thread_start_us":57,"threads_started":1}
I20260924 15:20:57.422477 12684 tablet_service.cc:1467] Processing CreateTablet for tablet 158caa1bae9f431abe24af89f334e557 (DEFAULT_TABLE table=test-table [id=709c339239bf4554ae8f28113f942085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:20:57.422714 12801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:57.404827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:43637 (local address 127.0.0.1:46614)
0924 15:20:57.406008 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:57.406053 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 15:20:57.406116 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:20:57.418893 (+ 12777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:20:57.418894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:20:57.418896 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:20:57.418931 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:57.418933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.420157 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.420158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:57.421560 (+ 1402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:57.421562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.422426 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.422426 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:57.422486 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:57.422491 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:20:57.422540 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 15:20:57.422578 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1152,"thread_start_us":109,"threads_started":1}
I20260924 15:20:57.422857 12684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 158caa1bae9f431abe24af89f334e557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:20:57.423179 12798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:57.404942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46614 (local address 127.31.12.65:43637)
0924 15:20:57.418731 (+ 13789us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:57.418736 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:57.418746 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:20:57.418777 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:57.418779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:57.418794 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:57.418999 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:20:57.419079 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.420068 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.421653 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.422301 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.422422 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:57.422483 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:57.423057 (+ 574us) server_negotiation.cc:299] Negotiation successful
0924 15:20:57.423104 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13747,"thread_start_us":41,"threads_started":1}
I20260924 15:20:57.424005 12426 tablet_service.cc:1467] Processing CreateTablet for tablet 158caa1bae9f431abe24af89f334e557 (DEFAULT_TABLE table=test-table [id=709c339239bf4554ae8f28113f942085]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:20:57.424294 12426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 158caa1bae9f431abe24af89f334e557. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:20:57.429615 12805 tablet_bootstrap.cc:492] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b: Bootstrap starting.
I20260924 15:20:57.430326 12805 tablet_bootstrap.cc:654] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b: Neither blocks nor log segments found. Creating new log.
I20260924 15:20:57.431026 12803 tablet_bootstrap.cc:492] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3: No bootstrap required, opened a new log
I20260924 15:20:57.431150 12803 ts_tablet_manager.cc:1397] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20260924 15:20:57.433640 12803 raft_consensus.cc:348] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e26ee780db54b87a1de08dd2890df8a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43637 } } peers { permanent_uuid: "ffac70834cbf4df7aa30a3acfcef060b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33055 } } peers { permanent_uuid: "dfc620b728c84aef988d21fe80f201d3" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35647 } }
I20260924 15:20:57.433941 12803 raft_consensus.cc:374] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:20:57.434033 12803 raft_consensus.cc:729] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfc620b728c84aef988d21fe80f201d3, State: Initialized, Role: FOLLOWER
I20260924 15:20:57.434314 12806 tablet_bootstrap.cc:492] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a: Bootstrap starting.
I20260924 15:20:57.434695 12805 log.cc:824] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b: Log is configured to *not* fsync() on all Append() calls
I20260924 15:20:57.434844 12803 consensus_queue.cc:260] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e26ee780db54b87a1de08dd2890df8a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43637 } } peers { permanent_uuid: "ffac70834cbf4df7aa30a3acfcef060b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33055 } } peers { permanent_uuid: "dfc620b728c84aef988d21fe80f201d3" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35647 } }
I20260924 15:20:57.435493 12803 ts_tablet_manager.cc:1428] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260924 15:20:57.436149 12806 tablet_bootstrap.cc:654] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a: Neither blocks nor log segments found. Creating new log.
I20260924 15:20:57.436313 12634 heartbeater.cc:503] Master 127.31.12.126:42033 was elected leader, sending a full tablet report...
I20260924 15:20:57.437480 12806 log.cc:824] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a: Log is configured to *not* fsync() on all Append() calls
I20260924 15:20:57.442653 12805 tablet_bootstrap.cc:492] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b: No bootstrap required, opened a new log
I20260924 15:20:57.442791 12805 ts_tablet_manager.cc:1397] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260924 15:20:57.445235 12805 raft_consensus.cc:348] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e26ee780db54b87a1de08dd2890df8a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43637 } } peers { permanent_uuid: "ffac70834cbf4df7aa30a3acfcef060b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33055 } } peers { permanent_uuid: "dfc620b728c84aef988d21fe80f201d3" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35647 } }
I20260924 15:20:57.445509 12805 raft_consensus.cc:374] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:20:57.445592 12805 raft_consensus.cc:729] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffac70834cbf4df7aa30a3acfcef060b, State: Initialized, Role: FOLLOWER
I20260924 15:20:57.445788 12805 consensus_queue.cc:260] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e26ee780db54b87a1de08dd2890df8a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43637 } } peers { permanent_uuid: "ffac70834cbf4df7aa30a3acfcef060b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33055 } } peers { permanent_uuid: "dfc620b728c84aef988d21fe80f201d3" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35647 } }
I20260924 15:20:57.446136 12805 ts_tablet_manager.cc:1428] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:20:57.446285 12769 heartbeater.cc:503] Master 127.31.12.126:42033 was elected leader, sending a full tablet report...
I20260924 15:20:57.447502 12806 tablet_bootstrap.cc:492] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a: No bootstrap required, opened a new log
I20260924 15:20:57.447569 12806 ts_tablet_manager.cc:1397] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.001s
I20260924 15:20:57.450273 12806 raft_consensus.cc:348] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e26ee780db54b87a1de08dd2890df8a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43637 } } peers { permanent_uuid: "ffac70834cbf4df7aa30a3acfcef060b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33055 } } peers { permanent_uuid: "dfc620b728c84aef988d21fe80f201d3" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35647 } }
I20260924 15:20:57.450491 12806 raft_consensus.cc:374] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:20:57.450569 12806 raft_consensus.cc:729] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e26ee780db54b87a1de08dd2890df8a, State: Initialized, Role: FOLLOWER
I20260924 15:20:57.450739 12806 consensus_queue.cc:260] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e26ee780db54b87a1de08dd2890df8a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43637 } } peers { permanent_uuid: "ffac70834cbf4df7aa30a3acfcef060b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33055 } } peers { permanent_uuid: "dfc620b728c84aef988d21fe80f201d3" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35647 } }
I20260924 15:20:57.451046 12806 ts_tablet_manager.cc:1428] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 15:20:57.451188 12494 heartbeater.cc:503] Master 127.31.12.126:42033 was elected leader, sending a full tablet report...
W20260924 15:20:57.479179 12495 tablet.cc:2369] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 15:20:57 dist-test-slave-zwbb krb5kdc[11278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263256, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for HTTP/127.31.12.84@KRBTEST.COM
I20260924 15:20:57.567271 12811 raft_consensus.cc:482] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:20:57.567417 12811 raft_consensus.cc:504] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e26ee780db54b87a1de08dd2890df8a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43637 } } peers { permanent_uuid: "ffac70834cbf4df7aa30a3acfcef060b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33055 } } peers { permanent_uuid: "dfc620b728c84aef988d21fe80f201d3" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35647 } }
I20260924 15:20:57.567879 12811 leader_election.cc:288] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ffac70834cbf4df7aa30a3acfcef060b (127.31.12.67:33055), dfc620b728c84aef988d21fe80f201d3 (127.31.12.66:35647)
I20260924 15:20:57.577543 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:57.568735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:57987 (local address 127.31.12.67:33055)
0924 15:20:57.568784 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:57.568786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:57.568793 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:20:57.568822 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:57.568823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:57.568832 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:57.568886 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:20:57.573997 (+ 5111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.574685 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.576496 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.577087 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.577140 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:57.577152 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:57.577397 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 15:20:57.577467 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 15:20:57.578102 12813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:57.568179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:33055 (local address 127.31.12.65:57987)
0924 15:20:57.568348 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:57.568358 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:20:57.568406 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:20:57.573813 (+ 5407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:20:57.573815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:20:57.573817 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:20:57.573909 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:57.573911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.575121 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.575122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:57.576383 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:57.576386 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.577241 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.577242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:57.577312 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:57.577316 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:20:57.577905 (+ 589us) client_negotiation.cc:240] Negotiation successful
0924 15:20:57.577968 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":113,"threads_started":1}
I20260924 15:20:57.578634 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158caa1bae9f431abe24af89f334e557" candidate_uuid: "1e26ee780db54b87a1de08dd2890df8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffac70834cbf4df7aa30a3acfcef060b" is_pre_election: true
I20260924 15:20:57.578805 12721 raft_consensus.cc:2454] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e26ee780db54b87a1de08dd2890df8a in term 0.
I20260924 15:20:57.579056 12379 leader_election.cc:302] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e26ee780db54b87a1de08dd2890df8a, ffac70834cbf4df7aa30a3acfcef060b; no voters:
I20260924 15:20:57.579190 12811 raft_consensus.cc:2781] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:20:57.579226 12811 raft_consensus.cc:482] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:20:57.579246 12811 raft_consensus.cc:3037] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:20:57.580135 12811 raft_consensus.cc:504] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e26ee780db54b87a1de08dd2890df8a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43637 } } peers { permanent_uuid: "ffac70834cbf4df7aa30a3acfcef060b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33055 } } peers { permanent_uuid: "dfc620b728c84aef988d21fe80f201d3" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35647 } }
I20260924 15:20:57.580265 12811 leader_election.cc:288] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a [CANDIDATE]: Term 1 election: Requested vote from peers ffac70834cbf4df7aa30a3acfcef060b (127.31.12.67:33055), dfc620b728c84aef988d21fe80f201d3 (127.31.12.66:35647)
I20260924 15:20:57.580669 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158caa1bae9f431abe24af89f334e557" candidate_uuid: "1e26ee780db54b87a1de08dd2890df8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffac70834cbf4df7aa30a3acfcef060b"
I20260924 15:20:57.580739 12721 raft_consensus.cc:3037] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:20:57.581681 12721 raft_consensus.cc:2454] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e26ee780db54b87a1de08dd2890df8a in term 1.
I20260924 15:20:57.581907 12379 leader_election.cc:302] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e26ee780db54b87a1de08dd2890df8a, ffac70834cbf4df7aa30a3acfcef060b; no voters:
I20260924 15:20:57.582008 12811 raft_consensus.cc:2781] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:20:57.582175 12811 raft_consensus.cc:686] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a [term 1 LEADER]: Becoming Leader. State: Replica: 1e26ee780db54b87a1de08dd2890df8a, State: Running, Role: LEADER
I20260924 15:20:57.582276 12811 consensus_queue.cc:237] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e26ee780db54b87a1de08dd2890df8a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43637 } } peers { permanent_uuid: "ffac70834cbf4df7aa30a3acfcef060b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33055 } } peers { permanent_uuid: "dfc620b728c84aef988d21fe80f201d3" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35647 } }
I20260924 15:20:57.583143 12292 catalog_manager.cc:5495] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e26ee780db54b87a1de08dd2890df8a (127.31.12.65). New cstate: current_term: 1 leader_uuid: "1e26ee780db54b87a1de08dd2890df8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e26ee780db54b87a1de08dd2890df8a" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffac70834cbf4df7aa30a3acfcef060b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfc620b728c84aef988d21fe80f201d3" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35647 } health_report { overall_health: UNKNOWN } } }
I20260924 15:20:57.585086 12802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:57.573422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:52365 (local address 127.31.12.66:35647)
0924 15:20:57.573516 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:57.573519 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:57.574075 (+ 556us) server_negotiation.cc:407] Connection header received
0924 15:20:57.574164 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:57.574166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:57.574174 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:57.574200 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:20:57.574374 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.575036 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.583878 (+ 8842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.584484 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.584576 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:57.584586 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:57.584958 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 15:20:57.585007 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 15:20:57.585462 12814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:57.573345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:35647 (local address 127.31.12.65:52365)
0924 15:20:57.574028 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:57.574037 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:20:57.574098 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:20:57.574254 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:20:57.574255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:20:57.574256 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:20:57.574309 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:57.574310 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.582731 (+ 8421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.582731 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:57.583753 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:57.583756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.584763 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.584764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:57.584863 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:57.584868 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:20:57.585319 (+ 451us) client_negotiation.cc:240] Negotiation successful
0924 15:20:57.585363 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":120,"threads_started":1}
I20260924 15:20:57.585947 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158caa1bae9f431abe24af89f334e557" candidate_uuid: "1e26ee780db54b87a1de08dd2890df8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfc620b728c84aef988d21fe80f201d3" is_pre_election: true
I20260924 15:20:57.586100 12587 raft_consensus.cc:2454] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e26ee780db54b87a1de08dd2890df8a in term 0.
I20260924 15:20:57.586199 12586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "158caa1bae9f431abe24af89f334e557" candidate_uuid: "1e26ee780db54b87a1de08dd2890df8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfc620b728c84aef988d21fe80f201d3"
I20260924 15:20:57.586249 12586 raft_consensus.cc:3037] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:20:57.587049 12586 raft_consensus.cc:2454] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e26ee780db54b87a1de08dd2890df8a in term 1.
I20260924 15:20:57.592249 12798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:57.589688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.31.12.65:43637)
0924 15:20:57.589726 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:57.589728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:57.589735 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:20:57.589763 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:57.589764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:57.589781 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:57.589849 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:20:57.590132 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.590682 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.591460 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:57.591648 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:57.591724 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:57.591928 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:20:57.592049 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:20:57.592134 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:57.592154 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 15:20:57.592174 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:20:57.594184 12586 raft_consensus.cc:1261] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3 [term 1 FOLLOWER]: Refusing update from remote peer 1e26ee780db54b87a1de08dd2890df8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:20:57.594272 12721 raft_consensus.cc:1261] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b [term 1 FOLLOWER]: Refusing update from remote peer 1e26ee780db54b87a1de08dd2890df8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:20:57.594592 12815 consensus_queue.cc:1035] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffac70834cbf4df7aa30a3acfcef060b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:20:57.594937 12811 consensus_queue.cc:1035] T 158caa1bae9f431abe24af89f334e557 P 1e26ee780db54b87a1de08dd2890df8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfc620b728c84aef988d21fe80f201d3" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 35647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:20:57.602082 12818 mvcc.cc:205] Tried to move back new op lower bound from 7332918303103152128 to 7332918303057481728. Current Snapshot: MvccSnapshot[applied={T|T < 7332918303103152128}]
I20260924 15:20:57.602815 12819 mvcc.cc:205] Tried to move back new op lower bound from 7332918303103152128 to 7332918303057481728. Current Snapshot: MvccSnapshot[applied={T|T < 7332918303103152128}]
I20260924 15:20:57.603663 12821 mvcc.cc:205] Tried to move back new op lower bound from 7332918303103152128 to 7332918303057481728. Current Snapshot: MvccSnapshot[applied={T|T < 7332918303103152128}]
I20260924 15:20:57.606140 12292 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:20:57.606227 12292 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:20:57.606973 12292 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 161898fcc0094d18be4a622c754412a4: Sending DeleteTablet for 3 replicas of tablet 158caa1bae9f431abe24af89f334e557
I20260924 15:20:57.607386 12426 tablet_service.cc:1514] Processing DeleteTablet for tablet 158caa1bae9f431abe24af89f334e557 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:20:57 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:46614
I20260924 15:20:57.607564 12684 tablet_service.cc:1514] Processing DeleteTablet for tablet 158caa1bae9f431abe24af89f334e557 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:20:57 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:35178
I20260924 15:20:57.607894 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet 158caa1bae9f431abe24af89f334e557 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:20:57 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:41944
I20260924 15:20:57.608472 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 12330
I20260924 15:20:57.616518 12829 tablet_replica.cc:330] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3: stopping tablet replica
I20260924 15:20:57.616197 12828 tablet_replica.cc:330] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b: stopping tablet replica
I20260924 15:20:57.617791 12828 raft_consensus.cc:2229] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:20:57.618191 12829 raft_consensus.cc:2229] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:20:57.618652 12829 raft_consensus.cc:2258] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:20:57.619057 12828 raft_consensus.cc:2258] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:20:57.621402 12829 ts_tablet_manager.cc:1905] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:20:57.623457 12828 ts_tablet_manager.cc:1905] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:20:57.625407 12828 ts_tablet_manager.cc:1918] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:20:57.625509 12828 log.cc:1196] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/158caa1bae9f431abe24af89f334e557
I20260924 15:20:57.625835 12828 ts_tablet_manager.cc:1939] T 158caa1bae9f431abe24af89f334e557 P ffac70834cbf4df7aa30a3acfcef060b: Deleting consensus metadata
I20260924 15:20:57.626302 12277 catalog_manager.cc:4841] TS ffac70834cbf4df7aa30a3acfcef060b (127.31.12.67:33055): tablet 158caa1bae9f431abe24af89f334e557 (table test-table [id=709c339239bf4554ae8f28113f942085]) successfully deleted
I20260924 15:20:57.626960 12829 ts_tablet_manager.cc:1918] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:20:57.627036 12829 log.cc:1196] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/158caa1bae9f431abe24af89f334e557
I20260924 15:20:57.627287 12829 ts_tablet_manager.cc:1939] T 158caa1bae9f431abe24af89f334e557 P dfc620b728c84aef988d21fe80f201d3: Deleting consensus metadata
I20260924 15:20:57.627805 12279 catalog_manager.cc:4841] TS dfc620b728c84aef988d21fe80f201d3 (127.31.12.66:35647): tablet 158caa1bae9f431abe24af89f334e557 (table test-table [id=709c339239bf4554ae8f28113f942085]) successfully deleted
W20260924 15:20:57.629047 12276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.12.65:43637 (error 108)
I20260924 15:20:57.629618 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 12496
I20260924 15:20:57.635578 12801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:57.635443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:43637 (local address 127.0.0.1:46624)
0924 15:20:57.635501 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:57.635542 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.65:43637: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 15:20:57.635952 12276 catalog_manager.cc:4584] TS 1e26ee780db54b87a1de08dd2890df8a (127.31.12.65:43637): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 158caa1bae9f431abe24af89f334e557: Network error: Client connection negotiation failed: client connection to 127.31.12.65:43637: connect: Connection refused (error 111)
I20260924 15:20:57.640120 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 12637
I20260924 15:20:57.646806 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 12258
I20260924 15:20:57.654915 31793 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 15:20:57.999390 31793 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 15:20:57.999490 31793 mini_ranger.cc:66] Stopping Ranger...
I20260924 15:20:58.343531 31793 mini_ranger.cc:68] Stopped Ranger
2026-09-24 15:20:58.343 UTC [11309] LOG: received smart shutdown request
2026-09-24 15:20:58.347 UTC [11309] LOG: background worker "logical replication launcher" (PID 11319) exited with exit code 1
2026-09-24 15:20:58.348 UTC [11313] LOG: shutting down
2026-09-24 15:20:58.501 UTC [11309] LOG: database system is shut down
2026-09-24T15:20:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46400 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:20:58 dist-test-slave-zwbb krb5kdc[12840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:20:58 dist-test-slave-zwbb krb5kdc[12840](info): set up 2 sockets
Sep 24 15:20:58 dist-test-slave-zwbb krb5kdc[12840](info): commencing operation
krb5kdc: starting...
W20260924 15:20:59.671129 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:20:59 dist-test-slave-zwbb krb5kdc[12840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:20:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:20:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:20:59.788010 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:38119
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:32903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:38119,127.31.12.125:33401,127.31.12.124:34725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:20:59.868590 12854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:20:59.868786 12854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:20:59.868808 12854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:20:59.869843 12854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:20:59.869877 12854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:20:59.869891 12854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:20:59.869903 12854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:20:59.869915 12854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:20:59.871320 12854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:32903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:38119,127.31.12.125:33401,127.31.12.124:34725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:38119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:20:59.871505 12854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:20:59.871676 12854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:20:59.876449 12859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:59.876416 12862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:20:59.876425 12860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:20:59.876688 12854 server_base.cc:1034] running on GCE node
I20260924 15:20:59.876920 12854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:20:59.877228 12854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:20:59.878399 12854 hybrid_clock.cc:648] HybridClock initialized: now 1790263259878364 us; error 53 us; skew 500 ppm
Sep 24 15:20:59 dist-test-slave-zwbb krb5kdc[12840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263259, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:20:59.884301 12854 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:20:59.884685 12854 webserver.cc:460] Webserver started at http://127.31.12.126:35557/ using document root <none> and password file <none>
I20260924 15:20:59.884881 12854 fs_manager.cc:359] Metadata directory not provided
I20260924 15:20:59.884922 12854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:20:59.885010 12854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:20:59.886080 12854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "284ab41cb81547a2a0d6b47c03a652df"
format_stamp: "Formatted at 2026-09-24 15:20:59 on dist-test-slave-zwbb"
server_key: "2ed9f7a4982beb9634245f508414980c"
server_key_iv: "4b1d6ef5c22be9fa37c8db3f8b22af24"
server_key_version: "encryptionkey@0"
I20260924 15:20:59.886396 12854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "284ab41cb81547a2a0d6b47c03a652df"
format_stamp: "Formatted at 2026-09-24 15:20:59 on dist-test-slave-zwbb"
server_key: "2ed9f7a4982beb9634245f508414980c"
server_key_iv: "4b1d6ef5c22be9fa37c8db3f8b22af24"
server_key_version: "encryptionkey@0"
I20260924 15:20:59.887657 12854 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:20:59.888350 12869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:20:59.888540 12854 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:20:59.888634 12854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "284ab41cb81547a2a0d6b47c03a652df"
format_stamp: "Formatted at 2026-09-24 15:20:59 on dist-test-slave-zwbb"
server_key: "2ed9f7a4982beb9634245f508414980c"
server_key_iv: "4b1d6ef5c22be9fa37c8db3f8b22af24"
server_key_version: "encryptionkey@0"
I20260924 15:20:59.888707 12854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:20:59.914247 12854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:20:59.916714 12854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:20:59.916879 12854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:20:59.920959 12854 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:38119
I20260924 15:20:59.920998 12923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:38119 every 8 connection(s)
I20260924 15:20:59.921382 12854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:20:59.922300 12924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:20:59.922977 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 12854
I20260924 15:20:59.923097 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:20:59.923293 31793 external_mini_cluster.cc:1428] Setting key 04f3dd8eb201c1bc1e0e757aae3eb226
I20260924 15:20:59.924652 12924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.12.125@KRBTEST.COM; defaulting to no policy
Sep 24 15:20:59 dist-test-slave-zwbb krb5kdc[12840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263259, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.125@KRBTEST.COM" created.
I20260924 15:20:59.934948 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:59.925587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38119 (local address 127.0.0.1:36384)
0924 15:20:59.925768 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:59.925807 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:20:59.926308 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:20:59.926603 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:20:59.926606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:20:59.926872 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:20:59.927088 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:59.927093 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:59.928100 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:59.928101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:59.928764 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:20:59.928766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:20:59.929266 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:20:59.929267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:20:59.929334 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:59.929868 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:20:59.929875 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:20:59.931398 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:20:59.933289 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:20:59.933296 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:20:59.933298 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:20:59.933607 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:20:59.933876 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:20:59.933876 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:20:59.933876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:20:59.933958 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:20:59.934404 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:20:59.934408 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:20:59.934472 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 15:20:59.934541 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 15:20:59.934685 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":97,"threads_started":1}
I20260924 15:20:59.934960 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:59.925588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36384 (local address 127.31.12.126:38119)
0924 15:20:59.925757 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 15:20:59.925766 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:20:59.925905 (+ 139us) server_negotiation.cc:407] Connection header received
0924 15:20:59.926466 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:20:59.926468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:20:59.926493 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:20:59.926597 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:20:59.927187 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:59.927982 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:59.928873 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:20:59.929172 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:20:59.929291 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:20:59.931484 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:20:59.931492 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:20:59.931492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:20:59.931498 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:20:59.933202 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:59.933689 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:59.933692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:59.933692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:59.933799 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:20:59.934033 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:20:59.934034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:20:59.934035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:20:59.934334 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:20:59.934421 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 15:20:59.934597 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 15:20:59.934770 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260924 15:20:59.935702 12924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:20:59.936071 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:59.935929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:33401 (local address 127.0.0.1:42242)
0924 15:20:59.935989 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:59.936039 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.125:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 15:20:59.936220 12871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.12.125:33401: connect: Connection refused (error 111)
I20260924 15:20:59.936669 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:59.936551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:33401 (local address 127.0.0.1:42258)
0924 15:20:59.936606 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:59.936637 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.125:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 15:20:59.936787 12924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.125:33401: Network error: Client connection negotiation failed: client connection to 127.31.12.125:33401: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.125.keytab.
Entry for principal kudu/127.31.12.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.125.keytab.
I20260924 15:20:59.948413 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.125:33401
--webserver_interface=127.31.12.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.125.keytab
--principal=kudu/127.31.12.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:32903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:38119,127.31.12.125:33401,127.31.12.124:34725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:20:59.986001 12924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } attempt: 1
I20260924 15:20:59.986599 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:59.986421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:33401 (local address 127.0.0.1:42262)
0924 15:20:59.986511 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:59.986563 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.125:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 15:20:59.987002 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:20:59.986910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:33401 (local address 127.0.0.1:42264)
0924 15:20:59.986956 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 15:20:59.986982 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.125:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 15:20:59.987181 12924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.125:33401: Network error: Client connection negotiation failed: client connection to 127.31.12.125:33401: connect: Connection refused (error 111)
W20260924 15:21:00.046049 12931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:00.046242 12931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:00.046264 12931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:00.047322 12931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:21:00.047355 12931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:00.047369 12931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:00.047382 12931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:21:00.047394 12931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:21:00.048975 12931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:32903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:38119,127.31.12.125:33401,127.31.12.124:34725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.125.keytab
--principal=kudu/127.31.12.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.125:33401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.12.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:00.049175 12931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:00.049335 12931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:00.053963 12939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:00.053992 12937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:00.053999 12936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:00.054255 12931 server_base.cc:1034] running on GCE node
I20260924 15:21:00.054509 12931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:00.054785 12931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:00.055321 12924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } attempt: 2
I20260924 15:21:00.055706 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.055568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:33401 (local address 127.0.0.1:42276)
0924 15:21:00.055628 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:00.055663 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.125:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 15:21:00.055933 12931 hybrid_clock.cc:648] HybridClock initialized: now 1790263260055927 us; error 35 us; skew 500 ppm
I20260924 15:21:00.056016 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.055929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:33401 (local address 127.0.0.1:42288)
0924 15:21:00.055972 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:00.055995 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.125:33401: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 15:21:00.056097 12924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.125:33401: Network error: Client connection negotiation failed: client connection to 127.31.12.125:33401: connect: Connection refused (error 111)
Sep 24 15:21:00 dist-test-slave-zwbb krb5kdc[12840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263260, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:00.061532 12931 init.cc:377] Logged in from keytab as kudu/127.31.12.125@KRBTEST.COM (short username kudu)
I20260924 15:21:00.061913 12931 webserver.cc:460] Webserver started at http://127.31.12.125:34989/ using document root <none> and password file <none>
I20260924 15:21:00.062117 12931 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:00.062158 12931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:00.062254 12931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:21:00.063233 12931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-1/data/instance:
uuid: "9e6971397e264e1e8112b0c2eef66239"
format_stamp: "Formatted at 2026-09-24 15:21:00 on dist-test-slave-zwbb"
server_key: "dfc39f774000140b365326c689eb67da"
server_key_iv: "70964b1928474049b5979cbed039fe28"
server_key_version: "encryptionkey@0"
I20260924 15:21:00.063534 12931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-1/wal/instance:
uuid: "9e6971397e264e1e8112b0c2eef66239"
format_stamp: "Formatted at 2026-09-24 15:21:00 on dist-test-slave-zwbb"
server_key: "dfc39f774000140b365326c689eb67da"
server_key_iv: "70964b1928474049b5979cbed039fe28"
server_key_version: "encryptionkey@0"
I20260924 15:21:00.064783 12931 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:21:00.065577 12946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:00.065776 12931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:21:00.065846 12931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-1/wal
uuid: "9e6971397e264e1e8112b0c2eef66239"
format_stamp: "Formatted at 2026-09-24 15:21:00 on dist-test-slave-zwbb"
server_key: "dfc39f774000140b365326c689eb67da"
server_key_iv: "70964b1928474049b5979cbed039fe28"
server_key_version: "encryptionkey@0"
I20260924 15:21:00.065938 12931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:00.089128 12931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:00.091432 12931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:00.091580 12931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:00.095625 12931 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.125:33401
I20260924 15:21:00.095662 12998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.125:33401 every 8 connection(s)
I20260924 15:21:00.096047 12931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-1/data/info.pb
I20260924 15:21:00.096889 12999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:21:00.098527 12999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:21:00.103420 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 12931
I20260924 15:21:00.103544 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-1/wal/instance
I20260924 15:21:00.103750 31793 external_mini_cluster.cc:1428] Setting key f5e9b55d6a2a3e211c790ceca3c14df0
Sep 24 15:21:00 dist-test-slave-zwbb krb5kdc[12840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263260, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.125@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:21:00.107712 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.099246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36396 (local address 127.31.12.126:38119)
0924 15:21:00.099314 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:00.099316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:00.099504 (+ 188us) server_negotiation.cc:407] Connection header received
0924 15:21:00.099905 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:00.099906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:00.099920 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:00.099996 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:00.100602 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.101098 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.102148 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.102357 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.102426 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.104497 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:00.104498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:00.104499 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:00.104508 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:00.106133 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.106657 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.106658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.106658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.106719 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.107044 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.107045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.107045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.107164 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:00.107225 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:00.107563 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 15:21:00.107648 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 15:21:00.107999 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.099184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38119 (local address 127.0.0.1:36396)
0924 15:21:00.099349 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:00.099390 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:00.099823 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:00.100087 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:00.100090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:00.100326 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:00.100530 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:00.100533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.101176 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.101177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:00.102041 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:00.102044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.102440 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.102441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:00.102540 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.103038 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:00.103046 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:00.104397 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:00.106227 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:00.106235 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:00.106237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:00.106572 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:00.106790 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:00.106791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:00.106791 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:00.106939 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:00.107237 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:00.107244 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:00.107416 (+ 172us) client_negotiation.cc:769] Sending connection context
0924 15:21:00.107536 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 15:21:00.107729 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260924 15:21:00.108646 12999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.12.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.124@KRBTEST.COM" created.
Sep 24 15:21:00 dist-test-slave-zwbb krb5kdc[12840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263260, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.125@KRBTEST.COM for kudu/127.31.12.125@KRBTEST.COM
I20260924 15:21:00.118515 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.108903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:33401 (local address 127.0.0.1:42302)
0924 15:21:00.108953 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:00.108964 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:00.109023 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:00.109283 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:00.109284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:00.109427 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:00.109521 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:00.109522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.110403 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.110404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:00.110906 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:00.110907 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.111379 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.111380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:00.111445 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.111870 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:00.111874 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:00.115464 (+ 3590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:00.117315 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:00.117316 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:00.117317 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:00.117602 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:00.117831 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:00.117832 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:00.117832 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:00.117888 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:00.118192 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:00.118192 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:00.118231 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 15:21:00.118286 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 15:21:00.118398 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 15:21:00.118618 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.108997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42302 (local address 127.31.12.125:33401)
0924 15:21:00.109156 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:00.109160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:00.109176 (+ 16us) server_negotiation.cc:407] Connection header received
0924 15:21:00.109192 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:00.109193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:00.109203 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:00.109283 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:00.109605 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.110300 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.111006 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.111293 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.111379 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.115565 (+ 4186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:00.115574 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:00.115575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:00.115583 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:00.117226 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.117702 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.117710 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.117711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.117762 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.117960 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.117961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.117961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.118112 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:00.118202 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:00.118325 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 15:21:00.118469 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260924 15:21:00.119592 12999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:21:00.119959 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.119825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:34725 (local address 127.0.0.1:55324)
0924 15:21:00.119872 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:00.119929 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:34725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 15:21:00.120086 12948 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.12.124:34725: connect: Connection refused (error 111)
I20260924 15:21:00.120492 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.120396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:34725 (local address 127.0.0.1:55328)
0924 15:21:00.120439 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:00.120463 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:34725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 15:21:00.120589 12999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.124:34725: Network error: Client connection negotiation failed: client connection to 127.31.12.124:34725: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.124.keytab.
Entry for principal kudu/127.31.12.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.124.keytab.
I20260924 15:21:00.131775 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.124:34725
--webserver_interface=127.31.12.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.124.keytab
--principal=kudu/127.31.12.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:32903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:38119,127.31.12.125:33401,127.31.12.124:34725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:21:00.147253 12924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } attempt: 3
Sep 24 15:21:00 dist-test-slave-zwbb krb5kdc[12840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263259, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for kudu/127.31.12.125@KRBTEST.COM
I20260924 15:21:00.154995 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.147651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:33401 (local address 127.0.0.1:42314)
0924 15:21:00.147746 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:00.147761 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:00.147828 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:00.148038 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:00.148040 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:00.148251 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:00.148318 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:00.148319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.149295 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.149296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:00.149900 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:00.149902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.150402 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.150403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:00.150486 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.151031 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:00.151036 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:00.152287 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:00.154006 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:00.154007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:00.154007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:00.154177 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:00.154382 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:00.154383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:00.154383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:00.154436 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:00.154703 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:00.154711 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:00.154754 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 15:21:00.154824 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 15:21:00.154907 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 15:21:00.155046 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.147736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42314 (local address 127.31.12.125:33401)
0924 15:21:00.147810 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:00.147812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:00.147826 (+ 14us) server_negotiation.cc:407] Connection header received
0924 15:21:00.147889 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:00.147890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:00.147904 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:00.147965 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:00.148397 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.149208 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.149979 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.150314 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.150387 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.152383 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:00.152385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:00.152386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:00.152394 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:00.153931 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.154246 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.154247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.154248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.154309 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.154494 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.154495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.154495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.154623 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:00.154690 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:00.154872 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 15:21:00.154957 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:21:00.155501 12924 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:21:00.155895 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.155788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:34725 (local address 127.0.0.1:55332)
0924 15:21:00.155842 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:00.155875 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:34725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 15:21:00.156245 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.156147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:34725 (local address 127.0.0.1:55346)
0924 15:21:00.156193 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:00.156229 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:34725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 15:21:00.156349 12924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.124:34725: Network error: Client connection negotiation failed: client connection to 127.31.12.124:34725: connect: Connection refused (error 111)
I20260924 15:21:00.169783 12999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } attempt: 1
I20260924 15:21:00.170213 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.170075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:34725 (local address 127.0.0.1:55360)
0924 15:21:00.170149 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:00.170185 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:34725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 15:21:00.170598 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.170511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:34725 (local address 127.0.0.1:55374)
0924 15:21:00.170555 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:00.170581 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:34725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 15:21:00.170709 12999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.124:34725: Network error: Client connection negotiation failed: client connection to 127.31.12.124:34725: connect: Connection refused (error 111)
I20260924 15:21:00.187495 12924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } attempt: 1
I20260924 15:21:00.188017 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.187877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:34725 (local address 127.0.0.1:55386)
0924 15:21:00.187946 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:00.187987 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:34725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260924 15:21:00.188407 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.188318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:34725 (local address 127.0.0.1:55392)
0924 15:21:00.188361 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:00.188391 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:34725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 15:21:00.188539 12924 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.124:34725: Network error: Client connection negotiation failed: client connection to 127.31.12.124:34725: connect: Connection refused (error 111)
W20260924 15:21:00.216015 13006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:00.216207 13006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:00.216228 13006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:00.217298 13006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:21:00.217331 13006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:00.217346 13006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:00.217358 13006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:21:00.217370 13006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:21:00.218884 13006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:32903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:38119,127.31.12.125:33401,127.31.12.124:34725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.124.keytab
--principal=kudu/127.31.12.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.124:34725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.12.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:00.219089 13006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:00.219264 13006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:00.223945 13012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:00.223968 13011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:00.224118 13006 server_base.cc:1034] running on GCE node
W20260924 15:21:00.223933 13014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:00.224468 13006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:00.224828 13006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:00.226001 13006 hybrid_clock.cc:648] HybridClock initialized: now 1790263260225972 us; error 39 us; skew 500 ppm
Sep 24 15:21:00 dist-test-slave-zwbb krb5kdc[12840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263260, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:00.228461 13006 init.cc:377] Logged in from keytab as kudu/127.31.12.124@KRBTEST.COM (short username kudu)
I20260924 15:21:00.228842 13006 webserver.cc:460] Webserver started at http://127.31.12.124:41879/ using document root <none> and password file <none>
I20260924 15:21:00.229040 13006 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:00.229081 13006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:00.229169 13006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:21:00.230237 13006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-2/data/instance:
uuid: "7c9a034ec3094d9e8b0184334e05b71b"
format_stamp: "Formatted at 2026-09-24 15:21:00 on dist-test-slave-zwbb"
server_key: "71503d5245f144c9c90cfd5c09506fe1"
server_key_iv: "983b12aa2a3993cec9c3eac689f42942"
server_key_version: "encryptionkey@0"
I20260924 15:21:00.230548 13006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-2/wal/instance:
uuid: "7c9a034ec3094d9e8b0184334e05b71b"
format_stamp: "Formatted at 2026-09-24 15:21:00 on dist-test-slave-zwbb"
server_key: "71503d5245f144c9c90cfd5c09506fe1"
server_key_iv: "983b12aa2a3993cec9c3eac689f42942"
server_key_version: "encryptionkey@0"
I20260924 15:21:00.231825 13006 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:21:00.232504 13021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:00.232697 13006 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:21:00.232782 13006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-2/wal
uuid: "7c9a034ec3094d9e8b0184334e05b71b"
format_stamp: "Formatted at 2026-09-24 15:21:00 on dist-test-slave-zwbb"
server_key: "71503d5245f144c9c90cfd5c09506fe1"
server_key_iv: "983b12aa2a3993cec9c3eac689f42942"
server_key_version: "encryptionkey@0"
I20260924 15:21:00.232836 13006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:00.238862 12999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } attempt: 2
I20260924 15:21:00.239264 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.239136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:34725 (local address 127.0.0.1:55408)
0924 15:21:00.239196 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:00.239233 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:34725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 15:21:00.239596 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.239510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:34725 (local address 127.0.0.1:55412)
0924 15:21:00.239554 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:00.239579 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:34725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 15:21:00.239696 12999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.124:34725: Network error: Client connection negotiation failed: client connection to 127.31.12.124:34725: connect: Connection refused (error 111)
I20260924 15:21:00.243289 13006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:00.244998 13006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:00.245111 13006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:00.249022 13006 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.124:34725
I20260924 15:21:00.249073 13075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.124:34725 every 8 connection(s)
I20260924 15:21:00.249395 13006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-2/data/info.pb
I20260924 15:21:00.250280 13076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:21:00.251768 13076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:21:00.257038 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 13006
I20260924 15:21:00.257117 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790263080303576-31793-0/minicluster-data/master-2/wal/instance
I20260924 15:21:00.257314 31793 external_mini_cluster.cc:1428] Setting key 5b7a17786fdb6ee3e326d776237a45cb
Sep 24 15:21:00 dist-test-slave-zwbb krb5kdc[12840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263260, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.124@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:21:00.261550 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.252447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36404 (local address 127.31.12.126:38119)
0924 15:21:00.252578 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:00.252581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:00.252754 (+ 173us) server_negotiation.cc:407] Connection header received
0924 15:21:00.253268 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:00.253269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:00.253282 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:00.253338 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:00.254002 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.254886 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.255874 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.256094 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.256183 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.258581 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:00.258582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:00.258582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:00.258588 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:00.260130 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.260552 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.260554 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.260554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.260617 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.260873 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.260875 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.260875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.261001 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:00.261073 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:00.261349 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 15:21:00.261464 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 15:21:00.261772 13078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.252415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38119 (local address 127.0.0.1:36404)
0924 15:21:00.252630 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:00.252666 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:00.253156 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:00.253409 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:00.253412 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:00.253710 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:00.253897 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:00.253902 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.254989 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.254990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:00.255775 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:00.255777 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.256193 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.256194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:00.256318 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.256750 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:00.256758 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:00.258489 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:00.260233 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:00.260241 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:00.260243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:00.260459 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:00.260687 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:00.260688 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:00.260688 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:00.260798 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:00.261075 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:00.261082 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:00.261215 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 15:21:00.261313 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 15:21:00.261492 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":87,"threads_started":1}
I20260924 15:21:00.262369 13076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:21:00 dist-test-slave-zwbb krb5kdc[12840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.124@KRBTEST.COM
I20260924 15:21:00.263700 12924 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } attempt: 2
Sep 24 15:21:00 dist-test-slave-zwbb krb5kdc[12840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263260, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.124@KRBTEST.COM for kudu/127.31.12.125@KRBTEST.COM
I20260924 15:21:00.266533 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.257762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55420 (local address 127.31.12.124:34725)
0924 15:21:00.257910 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:00.257914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:00.257927 (+ 13us) server_negotiation.cc:407] Connection header received
0924 15:21:00.258008 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:00.258009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:00.258058 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:00.258138 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:00.258846 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.259516 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.260243 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.260551 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.260630 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.263217 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:00.263224 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:00.263224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:00.263231 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:00.265247 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.265693 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.265695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.265696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.265744 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.265962 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.265962 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.265963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.266105 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:00.266297 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:00.266339 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 15:21:00.266412 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
Sep 24 15:21:00 dist-test-slave-zwbb krb5kdc[12840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263259, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for kudu/127.31.12.124@KRBTEST.COM
I20260924 15:21:00.270112 13078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.262619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:33401 (local address 127.0.0.1:42320)
0924 15:21:00.262658 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:00.262666 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:00.262697 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:00.262860 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:00.262860 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:00.262997 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:00.263041 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:00.263042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.263665 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.263666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:00.264365 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:00.264367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.264828 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.264829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:00.264912 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.265314 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:00.265319 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:00.266726 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:00.268648 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:00.268649 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:00.268649 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:00.268876 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:00.269098 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:00.269099 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:00.269099 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:00.269174 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:00.269754 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:00.269755 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:00.269807 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 15:21:00.269897 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 15:21:00.270001 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 15:21:00.270117 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.262700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42320 (local address 127.31.12.125:33401)
0924 15:21:00.262760 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:00.262762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:00.262769 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:21:00.262793 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:00.262795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:00.262804 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:00.262856 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:00.263129 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.263587 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.264443 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.264717 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.264778 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.266798 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:00.266799 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:00.266800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:00.266808 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:00.268575 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.268944 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.268945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.268946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.269004 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.269251 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.269252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.269252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.269673 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:00.269744 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:00.269932 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 15:21:00.270024 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 15:21:00.271005 13076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:21:00 dist-test-slave-zwbb krb5kdc[12840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:21:00.272895 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.263914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:34725 (local address 127.0.0.1:55436)
0924 15:21:00.263954 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:00.263963 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:00.264004 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:00.264505 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:00.264506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:00.264665 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:00.264702 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:00.264703 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.265600 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.265601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:00.266397 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:00.266401 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.266916 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.266917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:00.266983 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.267529 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:00.267534 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:00.268998 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:00.271867 (+ 2869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:00.271868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:00.271868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:00.272081 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:00.272304 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:00.272305 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:00.272305 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:00.272360 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:00.272621 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:00.272622 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:00.272662 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 15:21:00.272708 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 15:21:00.272799 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 15:21:00.273342 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.263956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.31.12.124:34725)
0924 15:21:00.264111 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:00.264113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:00.264122 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:21:00.264142 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:00.264143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:00.264153 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:00.264499 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:00.264782 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.265512 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.266504 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.266818 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.266908 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.269081 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:00.269082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:00.269083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:00.269089 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:00.271760 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.272163 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.272164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.272164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.272235 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.272420 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.272421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.272421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.272548 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:00.273074 (+ 526us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:00.273114 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 15:21:00.273195 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":112,"threads_started":1}
I20260924 15:21:00.275295 12924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df: Bootstrap starting.
Sep 24 15:21:00 dist-test-slave-zwbb krb5kdc[12840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263260, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.124@KRBTEST.COM for kudu/127.31.12.124@KRBTEST.COM
I20260924 15:21:00.275970 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.267535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36410 (local address 127.31.12.126:38119)
0924 15:21:00.267581 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:00.267583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:00.267590 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:21:00.267610 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:00.267612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:00.267621 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:00.267668 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:00.268329 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.269033 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.270032 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.270307 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.270388 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.272899 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:00.272900 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:00.272900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:00.272908 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:00.274670 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.275063 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.275064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.275064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.275119 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.275337 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.275337 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.275338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.275468 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:00.275621 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:00.275702 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 15:21:00.275780 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:21:00.276124 12924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df: Neither blocks nor log segments found. Creating new log.
I20260924 15:21:00.276471 12924 log.cc:824] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:00.277338 12924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df: No bootstrap required, opened a new log
I20260924 15:21:00.279083 13078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.271276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:34725 (local address 127.0.0.1:55440)
0924 15:21:00.271316 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:00.271326 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:00.271363 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:00.271490 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:00.271491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:00.271711 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:00.271736 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:00.271738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.272556 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.272557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:00.273568 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:00.273570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.274033 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.274034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:00.274096 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.274462 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:00.274467 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:00.275880 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:00.277937 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:00.277938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:00.277938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:00.278156 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:00.278366 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:00.278367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:00.278367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:00.278434 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:00.278747 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:00.278748 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:00.278793 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 15:21:00.278868 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 15:21:00.278961 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 15:21:00.279173 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.271311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55440 (local address 127.31.12.124:34725)
0924 15:21:00.271343 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:00.271345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:00.271352 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:21:00.271417 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:00.271418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:00.271427 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:00.271481 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:00.271811 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.272479 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.273663 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.273949 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.274018 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.275953 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:00.275955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:00.275955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:00.275961 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:00.277848 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.278237 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.278238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.278238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.278295 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.278507 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.278508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.278508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.278668 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:00.278737 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:00.278906 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 15:21:00.279039 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":19}
I20260924 15:21:00.279541 12924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ab41cb81547a2a0d6b47c03a652df" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } } peers { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } } peers { permanent_uuid: "7c9a034ec3094d9e8b0184334e05b71b" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } }
I20260924 15:21:00.279708 12924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:21:00.279749 12924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 284ab41cb81547a2a0d6b47c03a652df, State: Initialized, Role: FOLLOWER
I20260924 15:21:00.279881 12924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ab41cb81547a2a0d6b47c03a652df" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } } peers { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } } peers { permanent_uuid: "7c9a034ec3094d9e8b0184334e05b71b" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } }
I20260924 15:21:00.280194 12924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:21:00.281108 13076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b: Bootstrap starting.
Sep 24 15:21:00 dist-test-slave-zwbb krb5kdc[12840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.125@KRBTEST.COM
I20260924 15:21:00.281844 13076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b: Neither blocks nor log segments found. Creating new log.
I20260924 15:21:00.282162 13076 log.cc:824] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:00.283507 13076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b: No bootstrap required, opened a new log
W20260924 15:21:00.284826 13094 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260924 15:21:00.284898 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.276612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42336 (local address 127.31.12.125:33401)
0924 15:21:00.276653 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:00.276655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:00.276662 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:21:00.276681 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:00.276681 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:00.276689 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:00.276762 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:00.277403 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.278103 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.279033 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.279309 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.279343 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.281993 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:00.281995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:00.281995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:00.282003 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:00.283549 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.283942 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.283943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.283944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.283999 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.284196 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.284196 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.284197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.284319 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:00.284481 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:00.284721 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 15:21:00.284798 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 15:21:00.285137 13094 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:21:00.285377 13083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ab41cb81547a2a0d6b47c03a652df" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } } peers { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } } peers { permanent_uuid: "7c9a034ec3094d9e8b0184334e05b71b" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } } }
I20260924 15:21:00.285570 13083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:21:00.286561 13076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ab41cb81547a2a0d6b47c03a652df" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } } peers { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } } peers { permanent_uuid: "7c9a034ec3094d9e8b0184334e05b71b" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } }
I20260924 15:21:00.286775 13076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:21:00.286872 13076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c9a034ec3094d9e8b0184334e05b71b, State: Initialized, Role: FOLLOWER
I20260924 15:21:00.287047 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:21:00.287086 13076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ab41cb81547a2a0d6b47c03a652df" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } } peers { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } } peers { permanent_uuid: "7c9a034ec3094d9e8b0184334e05b71b" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } }
I20260924 15:21:00.287374 13076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:21:00.286134 12854 master_runner.cc:186] Error getting master registration for 127.31.12.125:33401: OK, instance_id { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" instance_seqno: 1790263260094528 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 15:21:00.289016 12854 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 15:21:00.289284 13112 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:21:00.289407 13112 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:21:00.289911 13097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ab41cb81547a2a0d6b47c03a652df" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } } peers { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } } peers { permanent_uuid: "7c9a034ec3094d9e8b0184334e05b71b" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } } }
I20260924 15:21:00.290063 13097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:21:00.290750 13006 master_runner.cc:186] Error getting master registration for 127.31.12.125:33401: OK, instance_id { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" instance_seqno: 1790263260094528 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 15:21:00.292924 13006 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 15:21:00.297086 12927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.289130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36424 (local address 127.31.12.126:38119)
0924 15:21:00.289173 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:00.289175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:00.289664 (+ 489us) server_negotiation.cc:407] Connection header received
0924 15:21:00.289777 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:00.289778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:00.289790 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:00.289823 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:00.290619 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.291377 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.292360 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.292672 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.292712 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.294256 (+ 1544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:00.294258 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:00.294258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:00.294266 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:00.295875 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.296255 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.296256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.296256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.296311 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.296502 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.296502 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.296503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.296631 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:00.296710 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:00.296911 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 15:21:00.296993 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:21:00.300660 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.289328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42344 (local address 127.31.12.125:33401)
0924 15:21:00.289367 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:00.289369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:00.290270 (+ 901us) server_negotiation.cc:407] Connection header received
0924 15:21:00.290387 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:00.290388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:00.290400 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:00.290429 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:00.291150 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.291813 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.293540 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.293870 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.293918 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.296947 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:00.296948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:00.296949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:00.296955 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:00.298703 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.299425 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.299426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.299427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.299488 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.299724 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.299725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.299725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.299873 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:00.299946 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:00.300159 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 15:21:00.300237 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:21:00.303082 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.289698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55452 (local address 127.31.12.124:34725)
0924 15:21:00.289733 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:00.289735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:00.292812 (+ 3077us) server_negotiation.cc:407] Connection header received
0924 15:21:00.292903 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:00.292904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:00.292914 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:00.292944 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:00.294418 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.295154 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.297243 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.297547 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.297589 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.300094 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:00.300095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:00.300096 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:00.300103 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:00.301817 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.302268 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.302269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.302269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.302334 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.302542 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.302543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.302544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.302658 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:00.302755 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:00.302911 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 15:21:00.302991 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:21:00.330868 12999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } attempt: 3
Sep 24 15:21:00 dist-test-slave-zwbb krb5kdc[12840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263260, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.125@KRBTEST.COM for kudu/127.31.12.124@KRBTEST.COM
I20260924 15:21:00.339506 13001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.331221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:34725 (local address 127.0.0.1:55458)
0924 15:21:00.331271 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:00.331283 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:00.331318 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:00.331464 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:00.331465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:00.331651 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:00.331705 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:00.331707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.332653 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.332654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:00.333497 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:00.333500 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.333983 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.333984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:00.334042 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.334550 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:00.334555 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:00.336163 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:00.338385 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:00.338386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:00.338387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:00.338618 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:00.338844 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:00.338845 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:00.338846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:00.338914 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:00.339185 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:00.339186 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:00.339234 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 15:21:00.339308 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 15:21:00.339402 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 15:21:00.339524 13080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:00.331279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55458 (local address 127.31.12.124:34725)
0924 15:21:00.331321 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:00.331324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:00.331360 (+ 36us) server_negotiation.cc:407] Connection header received
0924 15:21:00.331384 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:00.331384 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:00.331394 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:00.331445 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:00.331785 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.332553 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.333590 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:00.333902 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:00.333976 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:00.336249 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:00.336251 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:00.336251 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:00.336259 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:00.338287 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.338699 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.338700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.338700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.338761 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:00.338981 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:00.338981 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:00.338982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:00.339115 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:00.339184 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:00.339344 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 15:21:00.339419 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:21:00.341387 12999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239: Bootstrap starting.
I20260924 15:21:00.342186 12999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239: Neither blocks nor log segments found. Creating new log.
I20260924 15:21:00.342552 12999 log.cc:824] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:00.343644 12999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239: No bootstrap required, opened a new log
I20260924 15:21:00.345944 12999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ab41cb81547a2a0d6b47c03a652df" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } } peers { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } } peers { permanent_uuid: "7c9a034ec3094d9e8b0184334e05b71b" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } }
I20260924 15:21:00.346279 12999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:21:00.346365 12999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e6971397e264e1e8112b0c2eef66239, State: Initialized, Role: FOLLOWER
I20260924 15:21:00.346540 12999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ab41cb81547a2a0d6b47c03a652df" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } } peers { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } } peers { permanent_uuid: "7c9a034ec3094d9e8b0184334e05b71b" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } }
I20260924 15:21:00.346930 12999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 15:21:00.348917 13129 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:21:00.349043 13129 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:21:00.346889 13118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ab41cb81547a2a0d6b47c03a652df" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } } peers { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } } peers { permanent_uuid: "7c9a034ec3094d9e8b0184334e05b71b" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } } }
I20260924 15:21:00.350278 13118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:21:00.351667 12931 master_runner.cc:263] No leader master found from master 9e6971397e264e1e8112b0c2eef66239
I20260924 15:21:00.351711 12931 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 15:21:00.417686 12875 tablet.cc:2369] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:21:00.495738 13027 tablet.cc:2369] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:21:00.550508 13118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:21:00.550629 13118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ab41cb81547a2a0d6b47c03a652df" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } } peers { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } } peers { permanent_uuid: "7c9a034ec3094d9e8b0184334e05b71b" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } }
I20260924 15:21:00.550890 13118 leader_election.cc:288] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 284ab41cb81547a2a0d6b47c03a652df (127.31.12.126:38119), 7c9a034ec3094d9e8b0184334e05b71b (127.31.12.124:34725)
I20260924 15:21:00.551151 12896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9e6971397e264e1e8112b0c2eef66239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284ab41cb81547a2a0d6b47c03a652df" is_pre_election: true
I20260924 15:21:00.551280 12896 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e6971397e264e1e8112b0c2eef66239 in term 0.
I20260924 15:21:00.551220 13049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9e6971397e264e1e8112b0c2eef66239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c9a034ec3094d9e8b0184334e05b71b" is_pre_election: true
I20260924 15:21:00.551405 13049 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e6971397e264e1e8112b0c2eef66239 in term 0.
I20260924 15:21:00.551604 12948 leader_election.cc:302] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c9a034ec3094d9e8b0184334e05b71b, 9e6971397e264e1e8112b0c2eef66239; no voters:
I20260924 15:21:00.551735 13118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:21:00.551795 13118 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:21:00.551824 13118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:21:00.552578 13118 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ab41cb81547a2a0d6b47c03a652df" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } } peers { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } } peers { permanent_uuid: "7c9a034ec3094d9e8b0184334e05b71b" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } }
I20260924 15:21:00.552704 13118 leader_election.cc:288] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [CANDIDATE]: Term 1 election: Requested vote from peers 284ab41cb81547a2a0d6b47c03a652df (127.31.12.126:38119), 7c9a034ec3094d9e8b0184334e05b71b (127.31.12.124:34725)
I20260924 15:21:00.552873 12896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9e6971397e264e1e8112b0c2eef66239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "284ab41cb81547a2a0d6b47c03a652df"
I20260924 15:21:00.552901 13049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9e6971397e264e1e8112b0c2eef66239" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c9a034ec3094d9e8b0184334e05b71b"
I20260924 15:21:00.552956 12896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:21:00.553006 13049 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:21:00.553789 12896 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e6971397e264e1e8112b0c2eef66239 in term 1.
I20260924 15:21:00.553813 13049 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e6971397e264e1e8112b0c2eef66239 in term 1.
I20260924 15:21:00.553998 12948 leader_election.cc:302] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c9a034ec3094d9e8b0184334e05b71b, 9e6971397e264e1e8112b0c2eef66239; no voters:
I20260924 15:21:00.554111 13118 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:21:00.554301 13118 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [term 1 LEADER]: Becoming Leader. State: Replica: 9e6971397e264e1e8112b0c2eef66239, State: Running, Role: LEADER
I20260924 15:21:00.554458 13118 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ab41cb81547a2a0d6b47c03a652df" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } } peers { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } } peers { permanent_uuid: "7c9a034ec3094d9e8b0184334e05b71b" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } }
I20260924 15:21:00.554913 13131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e6971397e264e1e8112b0c2eef66239. Latest consensus state: current_term: 1 leader_uuid: "9e6971397e264e1e8112b0c2eef66239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ab41cb81547a2a0d6b47c03a652df" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } } peers { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } } peers { permanent_uuid: "7c9a034ec3094d9e8b0184334e05b71b" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } } }
I20260924 15:21:00.555001 13131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [sys.catalog]: This master's current role is: LEADER
I20260924 15:21:00.555194 13133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:21:00.555375 13133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:21:00.556818 12896 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df [term 1 FOLLOWER]: Refusing update from remote peer 9e6971397e264e1e8112b0c2eef66239: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:21:00.556818 13049 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b [term 1 FOLLOWER]: Refusing update from remote peer 9e6971397e264e1e8112b0c2eef66239: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:21:00.557179 13118 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c9a034ec3094d9e8b0184334e05b71b" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:21:00.557294 13131 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [LEADER]: Connected to new peer: Peer: permanent_uuid: "284ab41cb81547a2a0d6b47c03a652df" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:21:00.558521 13083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e6971397e264e1e8112b0c2eef66239. Latest consensus state: current_term: 1 leader_uuid: "9e6971397e264e1e8112b0c2eef66239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ab41cb81547a2a0d6b47c03a652df" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } } peers { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } } peers { permanent_uuid: "7c9a034ec3094d9e8b0184334e05b71b" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } } }
I20260924 15:21:00.558610 13083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:21:00.558650 13097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e6971397e264e1e8112b0c2eef66239. Latest consensus state: current_term: 1 leader_uuid: "9e6971397e264e1e8112b0c2eef66239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ab41cb81547a2a0d6b47c03a652df" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } } peers { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } } peers { permanent_uuid: "7c9a034ec3094d9e8b0184334e05b71b" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } } }
I20260924 15:21:00.558739 13097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:21:00.559489 13118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e6971397e264e1e8112b0c2eef66239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ab41cb81547a2a0d6b47c03a652df" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } } peers { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } } peers { permanent_uuid: "7c9a034ec3094d9e8b0184334e05b71b" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } } }
I20260924 15:21:00.559579 13118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [sys.catalog]: This master's current role is: LEADER
I20260924 15:21:00.559559 13131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e6971397e264e1e8112b0c2eef66239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ab41cb81547a2a0d6b47c03a652df" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } } peers { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } } peers { permanent_uuid: "7c9a034ec3094d9e8b0184334e05b71b" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } } }
I20260924 15:21:00.559613 13131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239 [sys.catalog]: This master's current role is: LEADER
I20260924 15:21:00.559641 13134 mvcc.cc:205] Tried to move back new op lower bound from 7332918315238068224 to 7332918315231739904. Current Snapshot: MvccSnapshot[applied={T|T < 7332918315238068224}]
I20260924 15:21:00.559701 13135 mvcc.cc:205] Tried to move back new op lower bound from 7332918315238068224 to 7332918315231739904. Current Snapshot: MvccSnapshot[applied={T|T < 7332918315238068224}]
I20260924 15:21:00.559701 13097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e6971397e264e1e8112b0c2eef66239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ab41cb81547a2a0d6b47c03a652df" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } } peers { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } } peers { permanent_uuid: "7c9a034ec3094d9e8b0184334e05b71b" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } } }
I20260924 15:21:00.559782 13097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:21:00.559782 13083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e6971397e264e1e8112b0c2eef66239" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "284ab41cb81547a2a0d6b47c03a652df" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38119 } } peers { permanent_uuid: "9e6971397e264e1e8112b0c2eef66239" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 33401 } } peers { permanent_uuid: "7c9a034ec3094d9e8b0184334e05b71b" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 34725 } } }
I20260924 15:21:00.559846 13083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:21:00.559952 13136 mvcc.cc:205] Tried to move back new op lower bound from 7332918315238068224 to 7332918315231739904. Current Snapshot: MvccSnapshot[applied={T|T < 7332918315238068224}]
I20260924 15:21:00.560407 13133 catalog_manager.cc:1347] Generated new cluster ID: a23841c4483e405f8b9b1b5c15b925ae
I20260924 15:21:00.560465 13133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:21:00.577750 13133 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:21:00.578365 13133 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:21:00.586719 13133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e6971397e264e1e8112b0c2eef66239: Generated new TSK 0
I20260924 15:21:00.586946 13133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:21:01.285768 13094 catalog_manager.cc:1259] Loaded cluster ID: a23841c4483e405f8b9b1b5c15b925ae
I20260924 15:21:01.285856 13094 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df: loading cluster ID for follower catalog manager: success
I20260924 15:21:01.287158 13094 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df: acquiring CA information for follower catalog manager: success
I20260924 15:21:01.287524 13094 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 284ab41cb81547a2a0d6b47c03a652df: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:21:01.289851 13112 catalog_manager.cc:1259] Loaded cluster ID: a23841c4483e405f8b9b1b5c15b925ae
I20260924 15:21:01.289901 13112 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b: loading cluster ID for follower catalog manager: success
I20260924 15:21:01.290666 13112 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b: acquiring CA information for follower catalog manager: success
I20260924 15:21:01.290930 13112 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7c9a034ec3094d9e8b0184334e05b71b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:21:01.643616 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 12854
I20260924 15:21:01.649490 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 12931
I20260924 15:21:01.654582 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 13006
2026-09-24T15:21:01Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3041 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:21:01 dist-test-slave-zwbb krb5kdc[13149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:21:01 dist-test-slave-zwbb krb5kdc[13149](info): set up 2 sockets
Sep 24 15:21:01 dist-test-slave-zwbb krb5kdc[13149](info): commencing operation
krb5kdc: starting...
W20260924 15:21:02.686962 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:21:02 dist-test-slave-zwbb krb5kdc[13149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:21:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:21:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:21:02.804005 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:40087
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:45821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:40087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:21:02.884979 13162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:02.885213 13162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:02.885251 13162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:02.886454 13162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:21:02.886519 13162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:02.886543 13162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:02.886564 13162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:21:02.886593 13162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:21:02.888062 13162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:45821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:40087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:40087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:02.888335 13162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:02.888559 13162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:02.893646 13170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:02.893621 13167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:02.893632 13168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:02.893661 13162 server_base.cc:1034] running on GCE node
I20260924 15:21:02.894043 13162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:02.894325 13162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:02.895469 13162 hybrid_clock.cc:648] HybridClock initialized: now 1790263262895448 us; error 36 us; skew 500 ppm
Sep 24 15:21:02 dist-test-slave-zwbb krb5kdc[13149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263262, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:02.898027 13162 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:21:02.898469 13162 webserver.cc:460] Webserver started at http://127.31.12.126:40799/ using document root <none> and password file <none>
I20260924 15:21:02.898718 13162 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:02.898787 13162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:02.898924 13162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:21:02.899941 13162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "fd8cf4a7699e43c7b391614f028877fc"
format_stamp: "Formatted at 2026-09-24 15:21:02 on dist-test-slave-zwbb"
server_key: "4808c9e67174d119103a12f22fcb1a15"
server_key_iv: "f6365710f5efab5b20a81754e27438a5"
server_key_version: "encryptionkey@0"
I20260924 15:21:02.900364 13162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "fd8cf4a7699e43c7b391614f028877fc"
format_stamp: "Formatted at 2026-09-24 15:21:02 on dist-test-slave-zwbb"
server_key: "4808c9e67174d119103a12f22fcb1a15"
server_key_iv: "f6365710f5efab5b20a81754e27438a5"
server_key_version: "encryptionkey@0"
I20260924 15:21:02.901742 13162 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:21:02.902484 13177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:02.902657 13162 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:21:02.902761 13162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "fd8cf4a7699e43c7b391614f028877fc"
format_stamp: "Formatted at 2026-09-24 15:21:02 on dist-test-slave-zwbb"
server_key: "4808c9e67174d119103a12f22fcb1a15"
server_key_iv: "f6365710f5efab5b20a81754e27438a5"
server_key_version: "encryptionkey@0"
I20260924 15:21:02.902844 13162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:02.911172 13162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:02.913076 13162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:02.913233 13162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:02.917162 13162 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:40087
I20260924 15:21:02.917198 13231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:40087 every 8 connection(s)
I20260924 15:21:02.917595 13162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:21:02.918437 13232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:21:02.919132 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 13162
I20260924 15:21:02.919253 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:21:02.919468 31793 external_mini_cluster.cc:1428] Setting key 6222e3cc5b5efb333a1038d805e1303f
I20260924 15:21:02.921766 13232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc: Bootstrap starting.
I20260924 15:21:02.922453 13232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc: Neither blocks nor log segments found. Creating new log.
I20260924 15:21:02.922771 13232 log.cc:824] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:02.923506 13232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc: No bootstrap required, opened a new log
I20260924 15:21:02.925868 13232 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8cf4a7699e43c7b391614f028877fc" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 40087 } }
I20260924 15:21:02.926003 13232 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:21:02.926026 13232 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd8cf4a7699e43c7b391614f028877fc, State: Initialized, Role: FOLLOWER
I20260924 15:21:02.926172 13232 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8cf4a7699e43c7b391614f028877fc" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 40087 } }
I20260924 15:21:02.926277 13232 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:21:02.926311 13232 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:21:02.926359 13232 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc [term 0 FOLLOWER]: Advancing to term 1
Sep 24 15:21:02 dist-test-slave-zwbb krb5kdc[13149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:21:02.927070 13232 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8cf4a7699e43c7b391614f028877fc" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 40087 } }
I20260924 15:21:02.927186 13232 leader_election.cc:302] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd8cf4a7699e43c7b391614f028877fc; no voters:
I20260924 15:21:02.927377 13232 leader_election.cc:288] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:21:02.927433 13237 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:21:02.927624 13237 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc [term 1 LEADER]: Becoming Leader. State: Replica: fd8cf4a7699e43c7b391614f028877fc, State: Running, Role: LEADER
I20260924 15:21:02.927732 13232 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:21:02.927809 13237 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8cf4a7699e43c7b391614f028877fc" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 40087 } }
I20260924 15:21:02.928138 13237 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd8cf4a7699e43c7b391614f028877fc. Latest consensus state: current_term: 1 leader_uuid: "fd8cf4a7699e43c7b391614f028877fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8cf4a7699e43c7b391614f028877fc" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 40087 } } }
I20260924 15:21:02.928175 13238 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd8cf4a7699e43c7b391614f028877fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd8cf4a7699e43c7b391614f028877fc" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 40087 } } }
I20260924 15:21:02.928294 13238 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc [sys.catalog]: This master's current role is: LEADER
I20260924 15:21:02.928496 13237 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc [sys.catalog]: This master's current role is: LEADER
I20260924 15:21:02.929641 13246 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:21:02.930342 13246 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:21:02.931208 13235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:02.919913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44082 (local address 127.31.12.126:40087)
0924 15:21:02.920098 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:02.920108 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:02.920153 (+ 45us) server_negotiation.cc:407] Connection header received
0924 15:21:02.920799 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:02.920810 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:02.920877 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:02.921068 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:02.921949 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:02.923306 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:02.924075 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:02.924706 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:02.924817 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:02.927506 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:02.927511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:02.927512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:02.927524 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:02.929253 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:02.929716 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:02.929720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:02.929721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:02.929842 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:02.930067 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:02.930068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:02.930068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:02.930386 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:02.930547 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:02.930830 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 15:21:02.930974 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":73,"threads_started":1}
I20260924 15:21:02.931939 13246 catalog_manager.cc:1347] Generated new cluster ID: 4841df701a0842ed8c780eebd9058527
I20260924 15:21:02.932001 13246 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:21:02.932473 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:21:02.951056 13246 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:21:02.951599 13246 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:21:02.956408 13246 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd8cf4a7699e43c7b391614f028877fc: Generated new TSK 0
I20260924 15:21:02.956605 13246 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:21:02.986354 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 13162
2026-09-24T15:21:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1330 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:21:03 dist-test-slave-zwbb krb5kdc[13261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:21:03 dist-test-slave-zwbb krb5kdc[13261](info): set up 2 sockets
Sep 24 15:21:03 dist-test-slave-zwbb krb5kdc[13261](info): commencing operation
krb5kdc: starting...
W20260924 15:21:04.039235 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:21:04 dist-test-slave-zwbb krb5kdc[13261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:21:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:21:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:21:04.153688 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:33699
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:33699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:21:04.233697 13274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:04.233943 13274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:04.233979 13274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:04.235117 13274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:21:04.235189 13274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:04.235213 13274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:04.235234 13274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:21:04.235263 13274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:21:04.236726 13274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:33699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:33699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:04.237008 13274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:04.237237 13274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:04.242118 13282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:04.242172 13280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:04.242290 13274 server_base.cc:1034] running on GCE node
W20260924 15:21:04.242117 13279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:04.242588 13274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:04.242889 13274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:04.244060 13274 hybrid_clock.cc:648] HybridClock initialized: now 1790263264244055 us; error 52 us; skew 500 ppm
Sep 24 15:21:04 dist-test-slave-zwbb krb5kdc[13261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263264, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:04.246649 13274 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:21:04.247130 13274 webserver.cc:460] Webserver started at http://127.31.12.126:38853/ using document root <none> and password file <none>
I20260924 15:21:04.247370 13274 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:04.247438 13274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:04.247599 13274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:21:04.248669 13274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "063f4028532a4ffbb3a633080866adbb"
format_stamp: "Formatted at 2026-09-24 15:21:04 on dist-test-slave-zwbb"
server_key: "8b7a2169c8d579cb00d442f2cafbb94a"
server_key_iv: "4b631f4e0dd583f40b452a05fe97e1b9"
server_key_version: "encryptionkey@0"
I20260924 15:21:04.249053 13274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "063f4028532a4ffbb3a633080866adbb"
format_stamp: "Formatted at 2026-09-24 15:21:04 on dist-test-slave-zwbb"
server_key: "8b7a2169c8d579cb00d442f2cafbb94a"
server_key_iv: "4b631f4e0dd583f40b452a05fe97e1b9"
server_key_version: "encryptionkey@0"
I20260924 15:21:04.250425 13274 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:21:04.251164 13289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:04.251332 13274 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:21:04.251441 13274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "063f4028532a4ffbb3a633080866adbb"
format_stamp: "Formatted at 2026-09-24 15:21:04 on dist-test-slave-zwbb"
server_key: "8b7a2169c8d579cb00d442f2cafbb94a"
server_key_iv: "4b631f4e0dd583f40b452a05fe97e1b9"
server_key_version: "encryptionkey@0"
I20260924 15:21:04.251516 13274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:04.260493 13274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:04.262725 13274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:04.262882 13274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:04.266927 13274 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:33699
I20260924 15:21:04.266963 13343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:33699 every 8 connection(s)
I20260924 15:21:04.267346 13274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:21:04.267982 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 13274
I20260924 15:21:04.268110 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:21:04.268317 31793 external_mini_cluster.cc:1428] Setting key a1500b43e2ff53e12afe68d8e0d19360
I20260924 15:21:04.268563 13344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:21:04.272261 13344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb: Bootstrap starting.
I20260924 15:21:04.273015 13344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb: Neither blocks nor log segments found. Creating new log.
I20260924 15:21:04.273326 13344 log.cc:824] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:04.274271 13344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb: No bootstrap required, opened a new log
Sep 24 15:21:04 dist-test-slave-zwbb krb5kdc[13261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:21:04.276803 13344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f4028532a4ffbb3a633080866adbb" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33699 } }
I20260924 15:21:04.276954 13344 raft_consensus.cc:374] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:21:04.277021 13344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063f4028532a4ffbb3a633080866adbb, State: Initialized, Role: FOLLOWER
I20260924 15:21:04.277168 13344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f4028532a4ffbb3a633080866adbb" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33699 } }
I20260924 15:21:04.277278 13344 raft_consensus.cc:388] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:21:04.277349 13344 raft_consensus.cc:482] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:21:04.277412 13344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:21:04.278004 13344 raft_consensus.cc:504] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f4028532a4ffbb3a633080866adbb" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33699 } }
I20260924 15:21:04.278098 13344 leader_election.cc:302] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 063f4028532a4ffbb3a633080866adbb; no voters:
I20260924 15:21:04.278318 13344 leader_election.cc:288] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:21:04.278358 13349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:21:04.278494 13349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 1 LEADER]: Becoming Leader. State: Replica: 063f4028532a4ffbb3a633080866adbb, State: Running, Role: LEADER
I20260924 15:21:04.278612 13344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:21:04.278671 13349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f4028532a4ffbb3a633080866adbb" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33699 } }
I20260924 15:21:04.279659 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:04.268991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45752 (local address 127.31.12.126:33699)
0924 15:21:04.269171 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:04.269181 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:04.269213 (+ 32us) server_negotiation.cc:407] Connection header received
0924 15:21:04.269804 (+ 591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:04.269811 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:04.269857 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:04.269968 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:04.270837 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.271698 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.272864 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.273302 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.273382 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:04.276186 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:04.276191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:04.276191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:04.276200 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:04.278030 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:04.278481 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:04.278484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:04.278484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:04.278594 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:04.278780 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:04.278781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:04.278781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:04.279043 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:04.279186 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:04.279310 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 15:21:04.279453 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260924 15:21:04.280045 13350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "063f4028532a4ffbb3a633080866adbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f4028532a4ffbb3a633080866adbb" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33699 } } }
I20260924 15:21:04.280220 13350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [sys.catalog]: This master's current role is: LEADER
I20260924 15:21:04.280164 13351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 063f4028532a4ffbb3a633080866adbb. Latest consensus state: current_term: 1 leader_uuid: "063f4028532a4ffbb3a633080866adbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f4028532a4ffbb3a633080866adbb" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33699 } } }
I20260924 15:21:04.280634 13351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [sys.catalog]: This master's current role is: LEADER
W20260924 15:21:04.280790 13363 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:21:04.280857 13363 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:21:04.280938 13364 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:21:04.281116 13364 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:21:04.286458 13364 catalog_manager.cc:1347] Generated new cluster ID: 52dd219a9bdf4a6a952d443fbed3ee85
I20260924 15:21:04.286522 13364 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:21:04.319082 13364 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:21:04.319638 13364 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:21:04.324990 13364 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb: Generated new TSK 0
I20260924 15:21:04.325192 13364 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
I20260924 15:21:04.356232 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:33699
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:21:04.446905 13371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:04.447103 13371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:04.447126 13371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:04.448191 13371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:04.448225 13371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:04.448249 13371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:21:04.449630 13371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:33699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:04.449806 13371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:04.449978 13371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:04.454859 13376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:04.454859 13379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:04.455085 13371 server_base.cc:1034] running on GCE node
W20260924 15:21:04.455148 13377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:04.455360 13371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:04.455612 13371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:04.456751 13371 hybrid_clock.cc:648] HybridClock initialized: now 1790263264456733 us; error 33 us; skew 500 ppm
Sep 24 15:21:04 dist-test-slave-zwbb krb5kdc[13261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263264, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:04.459184 13371 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:21:04.459610 13371 webserver.cc:460] Webserver started at http://127.31.12.65:41225/ using document root <none> and password file <none>
I20260924 15:21:04.459797 13371 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:04.459834 13371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:04.459918 13371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:21:04.460896 13371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "f9b26414cd9249ea8edceab73391e11e"
format_stamp: "Formatted at 2026-09-24 15:21:04 on dist-test-slave-zwbb"
server_key: "e4ac8195a6d6b85586c10e2dadfa66f0"
server_key_iv: "0f4086d05f8b305b891be96dcca01eb3"
server_key_version: "encryptionkey@0"
I20260924 15:21:04.461199 13371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "f9b26414cd9249ea8edceab73391e11e"
format_stamp: "Formatted at 2026-09-24 15:21:04 on dist-test-slave-zwbb"
server_key: "e4ac8195a6d6b85586c10e2dadfa66f0"
server_key_iv: "0f4086d05f8b305b891be96dcca01eb3"
server_key_version: "encryptionkey@0"
I20260924 15:21:04.462489 13371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:21:04.463138 13386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:04.463304 13371 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:21:04.463403 13371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "f9b26414cd9249ea8edceab73391e11e"
format_stamp: "Formatted at 2026-09-24 15:21:04 on dist-test-slave-zwbb"
server_key: "e4ac8195a6d6b85586c10e2dadfa66f0"
server_key_iv: "0f4086d05f8b305b891be96dcca01eb3"
server_key_version: "encryptionkey@0"
I20260924 15:21:04.463481 13371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:04.493055 13371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:04.495416 13371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:04.495579 13371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:04.495833 13371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:21:04.496155 13371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:21:04.496209 13371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:04.496241 13371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:21:04.496270 13371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:04.501842 13371 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:43305
I20260924 15:21:04.501878 13501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:43305 every 8 connection(s)
I20260924 15:21:04.502210 13371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:21:04 dist-test-slave-zwbb krb5kdc[13261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263264, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:21:04.511267 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:04.502889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:34959 (local address 127.31.12.126:33699)
0924 15:21:04.502939 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:04.502942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:04.503177 (+ 235us) server_negotiation.cc:407] Connection header received
0924 15:21:04.503999 (+ 822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:04.504000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:04.504030 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:04.504081 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:04.504992 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.505566 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.506567 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.506782 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.506868 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:04.508937 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:04.508938 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:04.508938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:04.508945 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:04.510178 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:04.510543 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:04.510544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:04.510544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:04.510591 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:04.510790 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:04.510791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:04.510791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:04.510881 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:04.510926 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:04.511142 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 15:21:04.511206 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:21:04.511406 13504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:04.502876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.65:34959)
0924 15:21:04.503039 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:04.503071 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:04.503841 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:04.504167 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:04.504171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:04.504681 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:04.504863 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:04.504868 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.505658 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.505660 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:04.506463 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:04.506465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.506879 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.506880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:04.507026 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:04.507497 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:04.507504 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:04.508850 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:04.510248 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:04.510255 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:04.510256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:04.510475 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:04.510647 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:04.510648 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:04.510648 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:04.510740 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:04.510931 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:04.510935 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:04.511038 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 15:21:04.511115 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 15:21:04.511222 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":88,"threads_started":1}
I20260924 15:21:04.511911 13502 heartbeater.cc:347] Connected to a master server at 127.31.12.126:33699
I20260924 15:21:04.512013 13502 heartbeater.cc:464] Registering TS with master...
I20260924 15:21:04.512012 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 13371
I20260924 15:21:04.512120 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:21:04.512202 13502 heartbeater.cc:511] Master 127.31.12.126:33699 requested a full tablet report, sending...
I20260924 15:21:04.512311 31793 external_mini_cluster.cc:1428] Setting key ce86abbf8cfc927faceb240787d04cda
I20260924 15:21:04.512738 13306 ts_manager.cc:194] Registered new tserver with Master: f9b26414cd9249ea8edceab73391e11e (127.31.12.65:43305)
I20260924 15:21:04.513424 13306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:34959
WARNING: no policy specified for kudu/127.31.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
I20260924 15:21:04.536552 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:33699
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:21:04.615407 13507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:04.615593 13507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:04.615617 13507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:04.616744 13507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:04.616786 13507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:04.616838 13507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:21:04.618299 13507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:33699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:04.618536 13507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:04.618760 13507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:04.623612 13512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:04.623651 13507 server_base.cc:1034] running on GCE node
W20260924 15:21:04.623612 13513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:04.623629 13516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:04.624058 13507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:04.624306 13507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:04.625509 13507 hybrid_clock.cc:648] HybridClock initialized: now 1790263264625484 us; error 32 us; skew 500 ppm
Sep 24 15:21:04 dist-test-slave-zwbb krb5kdc[13261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263264, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:04.627986 13507 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:21:04.628423 13507 webserver.cc:460] Webserver started at http://127.31.12.66:44903/ using document root <none> and password file <none>
I20260924 15:21:04.628619 13507 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:04.628659 13507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:04.628746 13507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:21:04.629783 13507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "23c27fce05f440d99056a2db20681a73"
format_stamp: "Formatted at 2026-09-24 15:21:04 on dist-test-slave-zwbb"
server_key: "18b601ae31e2afc2ff83b4390f2ed1d1"
server_key_iv: "66e56b913285f6720ef56593eadc4105"
server_key_version: "encryptionkey@0"
I20260924 15:21:04.630128 13507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "23c27fce05f440d99056a2db20681a73"
format_stamp: "Formatted at 2026-09-24 15:21:04 on dist-test-slave-zwbb"
server_key: "18b601ae31e2afc2ff83b4390f2ed1d1"
server_key_iv: "66e56b913285f6720ef56593eadc4105"
server_key_version: "encryptionkey@0"
I20260924 15:21:04.631364 13507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:21:04.632019 13523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:04.632206 13507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:21:04.632289 13507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "23c27fce05f440d99056a2db20681a73"
format_stamp: "Formatted at 2026-09-24 15:21:04 on dist-test-slave-zwbb"
server_key: "18b601ae31e2afc2ff83b4390f2ed1d1"
server_key_iv: "66e56b913285f6720ef56593eadc4105"
server_key_version: "encryptionkey@0"
I20260924 15:21:04.632342 13507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:04.649320 13507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:04.651139 13507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:04.651296 13507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:04.651536 13507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:21:04.651837 13507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:21:04.651870 13507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:04.651890 13507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:21:04.651937 13507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:04.657796 13507 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:33433
I20260924 15:21:04.657860 13638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:33433 every 8 connection(s)
I20260924 15:21:04.658207 13507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:21:04.661134 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 13507
I20260924 15:21:04.661247 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:21:04.661437 31793 external_mini_cluster.cc:1428] Setting key 329c2b841bc885e8d5a99e132504fbfb
Sep 24 15:21:04 dist-test-slave-zwbb krb5kdc[13261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263264, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:21:04.668358 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:04.658927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:34067 (local address 127.31.12.126:33699)
0924 15:21:04.659005 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:04.659006 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:04.659206 (+ 200us) server_negotiation.cc:407] Connection header received
0924 15:21:04.659733 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:04.659735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:04.659746 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:04.659812 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:04.660741 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.661273 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.662525 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.662719 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.662793 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:04.665080 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:04.665082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:04.665082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:04.665090 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:04.666870 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:04.667368 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:04.667370 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:04.667370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:04.667433 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:04.667675 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:04.667676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:04.667676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:04.667826 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:04.667901 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:04.668168 (+ 267us) server_negotiation.cc:299] Negotiation successful
0924 15:21:04.668250 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:21:04.668378 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:04.658916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.66:34067)
0924 15:21:04.659092 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:04.659127 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:04.659632 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:04.659904 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:04.659908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:04.660442 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:04.660633 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:04.660639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.661383 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.661384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:04.662425 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:04.662428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.662798 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.662799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:04.662930 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:04.663413 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:04.663421 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:04.664988 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:04.666949 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:04.666956 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:04.666958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:04.667273 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:04.667505 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:04.667505 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:04.667505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:04.667614 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:04.667892 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:04.667896 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:04.668019 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 15:21:04.668121 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 15:21:04.668230 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":87,"threads_started":1}
I20260924 15:21:04.668992 13639 heartbeater.cc:347] Connected to a master server at 127.31.12.126:33699
I20260924 15:21:04.669094 13639 heartbeater.cc:464] Registering TS with master...
I20260924 15:21:04.669272 13639 heartbeater.cc:511] Master 127.31.12.126:33699 requested a full tablet report, sending...
I20260924 15:21:04.669713 13306 ts_manager.cc:194] Registered new tserver with Master: 23c27fce05f440d99056a2db20681a73 (127.31.12.66:33433)
I20260924 15:21:04.670101 13306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:34067
WARNING: no policy specified for kudu/127.31.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
I20260924 15:21:04.688211 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:33699
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:21:04.772481 13644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:04.772745 13644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:04.772789 13644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:04.774103 13644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:04.774183 13644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:04.774227 13644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:21:04.775668 13644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:33699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:04.775954 13644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:04.776172 13644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:04.781275 13649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:04.781337 13652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:04.781271 13650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:04.781697 13644 server_base.cc:1034] running on GCE node
I20260924 15:21:04.782099 13644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:04.782392 13644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:04.783540 13644 hybrid_clock.cc:648] HybridClock initialized: now 1790263264783517 us; error 37 us; skew 500 ppm
Sep 24 15:21:04 dist-test-slave-zwbb krb5kdc[13261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263264, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:04.785974 13644 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:21:04.786432 13644 webserver.cc:460] Webserver started at http://127.31.12.67:39879/ using document root <none> and password file <none>
I20260924 15:21:04.786666 13644 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:04.786710 13644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:04.786852 13644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:21:04.787864 13644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "825a0a4a54e14c3dbcda29a6354c51b0"
format_stamp: "Formatted at 2026-09-24 15:21:04 on dist-test-slave-zwbb"
server_key: "a250e0d320f3a06e03e0309cb0ebe3ee"
server_key_iv: "6d5f6ab1f945218046111fcd2123bc9e"
server_key_version: "encryptionkey@0"
I20260924 15:21:04.788228 13644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "825a0a4a54e14c3dbcda29a6354c51b0"
format_stamp: "Formatted at 2026-09-24 15:21:04 on dist-test-slave-zwbb"
server_key: "a250e0d320f3a06e03e0309cb0ebe3ee"
server_key_iv: "6d5f6ab1f945218046111fcd2123bc9e"
server_key_version: "encryptionkey@0"
I20260924 15:21:04.789547 13644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:21:04.790254 13659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:04.790443 13644 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:21:04.790529 13644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "825a0a4a54e14c3dbcda29a6354c51b0"
format_stamp: "Formatted at 2026-09-24 15:21:04 on dist-test-slave-zwbb"
server_key: "a250e0d320f3a06e03e0309cb0ebe3ee"
server_key_iv: "6d5f6ab1f945218046111fcd2123bc9e"
server_key_version: "encryptionkey@0"
I20260924 15:21:04.790596 13644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:04.810461 13644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:04.812271 13644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:04.812426 13644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:04.812644 13644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:21:04.812958 13644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:21:04.812990 13644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:04.813010 13644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:21:04.813056 13644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:04.818795 13644 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:41053
I20260924 15:21:04.818840 13773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:41053 every 8 connection(s)
I20260924 15:21:04.819130 13644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:21:04.822875 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 13644
I20260924 15:21:04.822937 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:21:04.823118 31793 external_mini_cluster.cc:1428] Setting key 887acaf90ad98a4429ca1ab69ac1c9c4
Sep 24 15:21:04 dist-test-slave-zwbb krb5kdc[13261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263264, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:21:04.827893 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:04.819830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:45653 (local address 127.31.12.126:33699)
0924 15:21:04.819893 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:04.819895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:04.820072 (+ 177us) server_negotiation.cc:407] Connection header received
0924 15:21:04.820551 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:04.820552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:04.820581 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:04.820629 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:04.821560 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.822037 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.823084 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.823353 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.823438 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:04.825664 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:04.825665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:04.825666 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:04.825671 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:04.826804 (+ 1133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:04.827160 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:04.827161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:04.827161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:04.827209 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:04.827427 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:04.827427 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:04.827427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:04.827524 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:04.827573 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:04.827778 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 15:21:04.827832 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:21:04.828068 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:04.819791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.67:45653)
0924 15:21:04.819938 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:04.819992 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:04.820452 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:04.820697 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:04.820701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:04.821229 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:04.821406 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:04.821411 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.822123 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.822124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:04.822955 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:04.822958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.823444 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.823445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:04.823578 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:04.824068 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:04.824076 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:04.825586 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:04.826871 (+ 1285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:04.826878 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:04.826879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:04.827097 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:04.827267 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:04.827267 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:04.827268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:04.827368 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:04.827578 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:04.827582 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:04.827687 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 15:21:04.827765 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 15:21:04.827878 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":76,"threads_started":1}
I20260924 15:21:04.828569 13774 heartbeater.cc:347] Connected to a master server at 127.31.12.126:33699
I20260924 15:21:04.828644 13774 heartbeater.cc:464] Registering TS with master...
I20260924 15:21:04.828833 13774 heartbeater.cc:511] Master 127.31.12.126:33699 requested a full tablet report, sending...
I20260924 15:21:04.829208 13306 ts_manager.cc:194] Registered new tserver with Master: 825a0a4a54e14c3dbcda29a6354c51b0 (127.31.12.67:41053)
I20260924 15:21:04.829594 13306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:45653
I20260924 15:21:04.834730 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:21:04.841288 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:04.836097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45756 (local address 127.31.12.126:33699)
0924 15:21:04.836132 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:04.836133 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:04.836255 (+ 122us) server_negotiation.cc:407] Connection header received
0924 15:21:04.836292 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:04.836292 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:04.836300 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:04.836346 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:04.836848 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.837291 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.838019 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.838197 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.838248 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:04.839281 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:04.839282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:04.839282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:04.839288 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:04.840392 (+ 1104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:04.840690 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:04.840690 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:04.840691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:04.840743 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:04.840893 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:04.840894 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:04.840894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:04.841012 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:04.841085 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:04.841179 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 15:21:04.841231 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 15:21:04.842757 13306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:21:04.844256 13306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:21:04.850421 13788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:04.846473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:33433 (local address 127.0.0.1:60780)
0924 15:21:04.847054 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:04.847082 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:04.847166 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:04.847379 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:04.847382 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:04.847385 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:04.847449 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:04.847451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.848238 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.848238 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:04.849267 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:04.849270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.850137 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.850138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:04.850220 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:04.850232 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 15:21:04.850301 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 15:21:04.850353 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":65,"threads_started":1}
I20260924 15:21:04.850435 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:04.846571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60780 (local address 127.31.12.66:33433)
0924 15:21:04.846818 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:04.846823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:04.847161 (+ 338us) server_negotiation.cc:407] Connection header received
0924 15:21:04.847273 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:04.847276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:04.847291 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:04.847359 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:04.847504 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.848167 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.849334 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.850053 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.850137 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:04.850197 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:04.850322 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 15:21:04.850364 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":104,"threads_started":1}
I20260924 15:21:04.851511 13572 tablet_service.cc:1467] Processing CreateTablet for tablet cc84ff9bce5d476ca5bf9daf6c62afee (DEFAULT_TABLE table=test-table [id=09e7d101148640afa58a193be8431d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:21:04.851889 13572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc84ff9bce5d476ca5bf9daf6c62afee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:21:04.851910 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:04.846802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:41053 (local address 127.0.0.1:35798)
0924 15:21:04.848303 (+ 1501us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:04.848311 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:04.848353 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:04.848502 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:04.848503 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:04.848504 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:04.848571 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:04.848573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.849533 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.849533 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:04.850649 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:04.850651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.851687 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.851688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:04.851760 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:04.851763 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:21:04.851806 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 15:21:04.851844 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1487,"thread_start_us":54,"threads_started":1}
I20260924 15:21:04.852141 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:04.846889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35798 (local address 127.31.12.67:41053)
0924 15:21:04.847081 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:04.847085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:04.848370 (+ 1285us) server_negotiation.cc:407] Connection header received
0924 15:21:04.848406 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:04.848408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:04.848429 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:04.848485 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:04.848637 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.849397 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.850752 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.851540 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.851610 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:04.851654 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:04.851997 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 15:21:04.852052 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
I20260924 15:21:04.852945 13708 tablet_service.cc:1467] Processing CreateTablet for tablet cc84ff9bce5d476ca5bf9daf6c62afee (DEFAULT_TABLE table=test-table [id=09e7d101148640afa58a193be8431d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:21:04.853124 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:04.846659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:43305 (local address 127.0.0.1:52718)
0924 15:21:04.848167 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:04.848177 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:04.848234 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:04.849618 (+ 1384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:04.849619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:04.849621 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:04.849675 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:04.849677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.850981 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.850982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:04.852037 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:04.852040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.852882 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.852883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:04.852954 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:04.852958 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:21:04.853021 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 15:21:04.853057 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1496,"thread_start_us":66,"threads_started":1}
I20260924 15:21:04.853159 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:04.847975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52718 (local address 127.31.12.65:43305)
0924 15:21:04.848141 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:04.848146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:04.849361 (+ 1215us) server_negotiation.cc:407] Connection header received
0924 15:21:04.849409 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:04.849412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:04.849428 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:04.849529 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:04.849761 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.850873 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.852128 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.852791 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.852866 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:04.852917 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:04.853045 (+ 128us) server_negotiation.cc:299] Negotiation successful
0924 15:21:04.853086 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":100,"threads_started":1}
I20260924 15:21:04.853267 13708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc84ff9bce5d476ca5bf9daf6c62afee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:21:04.853789 13434 tablet_service.cc:1467] Processing CreateTablet for tablet cc84ff9bce5d476ca5bf9daf6c62afee (DEFAULT_TABLE table=test-table [id=09e7d101148640afa58a193be8431d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:21:04.854095 13434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc84ff9bce5d476ca5bf9daf6c62afee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:21:04.855432 13791 tablet_bootstrap.cc:492] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73: Bootstrap starting.
I20260924 15:21:04.856099 13791 tablet_bootstrap.cc:654] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73: Neither blocks nor log segments found. Creating new log.
I20260924 15:21:04.856243 13792 tablet_bootstrap.cc:492] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0: Bootstrap starting.
I20260924 15:21:04.856848 13791 log.cc:824] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:04.856871 13792 tablet_bootstrap.cc:654] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0: Neither blocks nor log segments found. Creating new log.
I20260924 15:21:04.857066 13793 tablet_bootstrap.cc:492] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e: Bootstrap starting.
I20260924 15:21:04.857457 13792 log.cc:824] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:04.857714 13793 tablet_bootstrap.cc:654] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e: Neither blocks nor log segments found. Creating new log.
I20260924 15:21:04.858125 13791 tablet_bootstrap.cc:492] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73: No bootstrap required, opened a new log
I20260924 15:21:04.858172 13792 tablet_bootstrap.cc:492] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0: No bootstrap required, opened a new log
I20260924 15:21:04.858251 13792 ts_tablet_manager.cc:1397] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:21:04.858211 13791 ts_tablet_manager.cc:1397] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 15:21:04.858337 13793 log.cc:824] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:04.859087 13793 tablet_bootstrap.cc:492] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e: No bootstrap required, opened a new log
I20260924 15:21:04.859158 13793 ts_tablet_manager.cc:1397] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:21:04.859854 13792 raft_consensus.cc:348] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } } peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } }
I20260924 15:21:04.860010 13792 raft_consensus.cc:374] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:21:04.860074 13792 raft_consensus.cc:729] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825a0a4a54e14c3dbcda29a6354c51b0, State: Initialized, Role: FOLLOWER
I20260924 15:21:04.860199 13792 consensus_queue.cc:260] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } } peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } }
I20260924 15:21:04.860356 13791 raft_consensus.cc:348] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } } peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } }
I20260924 15:21:04.860493 13774 heartbeater.cc:503] Master 127.31.12.126:33699 was elected leader, sending a full tablet report...
I20260924 15:21:04.860507 13791 raft_consensus.cc:374] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:21:04.860491 13792 ts_tablet_manager.cc:1428] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:21:04.860556 13791 raft_consensus.cc:729] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c27fce05f440d99056a2db20681a73, State: Initialized, Role: FOLLOWER
I20260924 15:21:04.860661 13791 consensus_queue.cc:260] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } } peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } }
I20260924 15:21:04.860925 13791 ts_tablet_manager.cc:1428] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 15:21:04.860957 13639 heartbeater.cc:503] Master 127.31.12.126:33699 was elected leader, sending a full tablet report...
I20260924 15:21:04.861342 13793 raft_consensus.cc:348] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } } peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } }
I20260924 15:21:04.861558 13793 raft_consensus.cc:374] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:21:04.861619 13793 raft_consensus.cc:729] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b26414cd9249ea8edceab73391e11e, State: Initialized, Role: FOLLOWER
I20260924 15:21:04.861765 13793 consensus_queue.cc:260] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } } peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } }
I20260924 15:21:04.862088 13793 ts_tablet_manager.cc:1428] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 15:21:04.862092 13502 heartbeater.cc:503] Master 127.31.12.126:33699 was elected leader, sending a full tablet report...
W20260924 15:21:04.908785 13640 tablet.cc:2369] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:21:04.996608 13797 raft_consensus.cc:482] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:21:04.996726 13797 raft_consensus.cc:504] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } } peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } }
I20260924 15:21:04.997040 13797 leader_election.cc:288] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23c27fce05f440d99056a2db20681a73 (127.31.12.66:33433), f9b26414cd9249ea8edceab73391e11e (127.31.12.65:43305)
I20260924 15:21:05.000862 13786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:04.997331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:48495 (local address 127.31.12.66:33433)
0924 15:21:04.997385 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:04.997387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:04.997401 (+ 14us) server_negotiation.cc:407] Connection header received
0924 15:21:04.997453 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:04.997455 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:04.997466 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:04.997526 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:04.997727 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.998500 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.999927 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.000490 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.000556 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.000564 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:05.000758 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 15:21:05.000795 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 15:21:05.000862 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:04.997471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:54599 (local address 127.31.12.65:43305)
0924 15:21:04.997520 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:04.997522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:04.997668 (+ 146us) server_negotiation.cc:407] Connection header received
0924 15:21:04.997716 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:04.997717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:04.997727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:04.997785 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:04.997927 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.998592 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.999926 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.000483 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.000556 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.000564 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:05.000758 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 15:21:05.000795 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 15:21:05.000880 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:04.997229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:33433 (local address 127.31.12.67:48495)
0924 15:21:04.997288 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:04.997300 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:04.997375 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:04.997568 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:04.997569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:04.997571 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:04.997647 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:04.997650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.998578 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.998579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:04.999847 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:04.999848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.000570 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.000571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.000661 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.000663 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:21:05.000716 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 15:21:05.000771 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":53}
I20260924 15:21:05.000885 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:04.997340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:43305 (local address 127.31.12.67:54599)
0924 15:21:04.997583 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:04.997592 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:04.997652 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:04.997795 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:04.997796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:04.997797 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:04.997854 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:04.997856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:04.998665 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:04.998665 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:04.999832 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:04.999836 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.000570 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.000571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.000656 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.000660 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:21:05.000716 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 15:21:05.000771 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":20,"thread_start_us":182,"threads_started":1}
I20260924 15:21:05.001330 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc84ff9bce5d476ca5bf9daf6c62afee" candidate_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23c27fce05f440d99056a2db20681a73" is_pre_election: true
I20260924 15:21:05.001526 13592 raft_consensus.cc:2454] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 825a0a4a54e14c3dbcda29a6354c51b0 in term 0.
I20260924 15:21:05.001537 13455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc84ff9bce5d476ca5bf9daf6c62afee" candidate_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b26414cd9249ea8edceab73391e11e" is_pre_election: true
I20260924 15:21:05.001662 13455 raft_consensus.cc:2454] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 825a0a4a54e14c3dbcda29a6354c51b0 in term 0.
I20260924 15:21:05.001770 13663 leader_election.cc:302] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23c27fce05f440d99056a2db20681a73, 825a0a4a54e14c3dbcda29a6354c51b0; no voters:
I20260924 15:21:05.001930 13797 raft_consensus.cc:2781] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:21:05.002023 13797 raft_consensus.cc:482] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:21:05.002043 13797 raft_consensus.cc:3037] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:21:05.002764 13797 raft_consensus.cc:504] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } } peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } }
I20260924 15:21:05.002897 13797 leader_election.cc:288] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [CANDIDATE]: Term 1 election: Requested vote from peers 23c27fce05f440d99056a2db20681a73 (127.31.12.66:33433), f9b26414cd9249ea8edceab73391e11e (127.31.12.65:43305)
W20260924 15:21:05.002914 13503 tablet.cc:2369] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:21:05.003098 13592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc84ff9bce5d476ca5bf9daf6c62afee" candidate_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23c27fce05f440d99056a2db20681a73"
I20260924 15:21:05.003108 13455 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc84ff9bce5d476ca5bf9daf6c62afee" candidate_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b26414cd9249ea8edceab73391e11e"
I20260924 15:21:05.003181 13455 raft_consensus.cc:3037] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:21:05.003181 13592 raft_consensus.cc:3037] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:21:05.004066 13592 raft_consensus.cc:2454] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 825a0a4a54e14c3dbcda29a6354c51b0 in term 1.
I20260924 15:21:05.004066 13455 raft_consensus.cc:2454] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 825a0a4a54e14c3dbcda29a6354c51b0 in term 1.
I20260924 15:21:05.004215 13663 leader_election.cc:302] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 825a0a4a54e14c3dbcda29a6354c51b0, f9b26414cd9249ea8edceab73391e11e; no voters:
I20260924 15:21:05.004312 13797 raft_consensus.cc:2781] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:21:05.004459 13797 raft_consensus.cc:686] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [term 1 LEADER]: Becoming Leader. State: Replica: 825a0a4a54e14c3dbcda29a6354c51b0, State: Running, Role: LEADER
I20260924 15:21:05.004561 13797 consensus_queue.cc:237] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } } peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } }
I20260924 15:21:05.005551 13305 catalog_manager.cc:5495] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 825a0a4a54e14c3dbcda29a6354c51b0 (127.31.12.67). New cstate: current_term: 1 leader_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } health_report { overall_health: HEALTHY } } }
I20260924 15:21:05.034092 13789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.031637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.31.12.67:41053)
0924 15:21:05.031707 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:05.031709 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:05.031718 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:21:05.031764 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:05.031765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:05.031782 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:05.031855 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:21:05.032011 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.032473 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.033264 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.033468 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.033553 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.033735 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:21:05.033843 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:21:05.033939 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:05.033958 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 15:21:05.033985 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 15:21:05.036026 13455 raft_consensus.cc:1261] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Refusing update from remote peer 825a0a4a54e14c3dbcda29a6354c51b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:21:05.036046 13592 raft_consensus.cc:1261] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73 [term 1 FOLLOWER]: Refusing update from remote peer 825a0a4a54e14c3dbcda29a6354c51b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:21:05.036386 13797 consensus_queue.cc:1035] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:21:05.036530 13801 consensus_queue.cc:1035] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:21:05.038827 13805 mvcc.cc:205] Tried to move back new op lower bound from 7332918333584650240 to 7332918333459349504. Current Snapshot: MvccSnapshot[applied={T|T < 7332918333584650240}]
I20260924 15:21:05.041599 13305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:21:05.041680 13305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:21:05.042361 13305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb: Sending DeleteTablet for 3 replicas of tablet cc84ff9bce5d476ca5bf9daf6c62afee
I20260924 15:21:05.042707 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 13371
I20260924 15:21:05.042737 13434 tablet_service.cc:1514] Processing DeleteTablet for tablet cc84ff9bce5d476ca5bf9daf6c62afee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:21:05 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:52718
I20260924 15:21:05.042816 13572 tablet_service.cc:1514] Processing DeleteTablet for tablet cc84ff9bce5d476ca5bf9daf6c62afee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:21:05 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:60780
I20260924 15:21:05.043021 13708 tablet_service.cc:1514] Processing DeleteTablet for tablet cc84ff9bce5d476ca5bf9daf6c62afee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:21:05 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:35798
I20260924 15:21:05.044121 13816 tablet_replica.cc:330] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0: stopping tablet replica
I20260924 15:21:05.044041 13815 tablet_replica.cc:330] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73: stopping tablet replica
I20260924 15:21:05.044406 13815 raft_consensus.cc:2229] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:21:05.044358 13816 raft_consensus.cc:2229] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:21:05.044641 13815 raft_consensus.cc:2258] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:21:05.044839 13816 raft_consensus.cc:2258] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:21:05.045392 13815 ts_tablet_manager.cc:1905] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:21:05.045548 13816 ts_tablet_manager.cc:1905] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:21:05.047268 13816 ts_tablet_manager.cc:1918] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:21:05.047333 13816 log.cc:1196] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/cc84ff9bce5d476ca5bf9daf6c62afee
I20260924 15:21:05.047482 13815 ts_tablet_manager.cc:1918] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:21:05.047559 13815 log.cc:1196] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/cc84ff9bce5d476ca5bf9daf6c62afee
I20260924 15:21:05.047634 13816 ts_tablet_manager.cc:1939] T cc84ff9bce5d476ca5bf9daf6c62afee P 825a0a4a54e14c3dbcda29a6354c51b0: Deleting consensus metadata
I20260924 15:21:05.047835 13815 ts_tablet_manager.cc:1939] T cc84ff9bce5d476ca5bf9daf6c62afee P 23c27fce05f440d99056a2db20681a73: Deleting consensus metadata
I20260924 15:21:05.048075 13293 catalog_manager.cc:4841] TS 825a0a4a54e14c3dbcda29a6354c51b0 (127.31.12.67:41053): tablet cc84ff9bce5d476ca5bf9daf6c62afee (table test-table [id=09e7d101148640afa58a193be8431d7c]) successfully deleted
I20260924 15:21:05.048239 13293 catalog_manager.cc:4841] TS 23c27fce05f440d99056a2db20681a73 (127.31.12.66:33433): tablet cc84ff9bce5d476ca5bf9daf6c62afee (table test-table [id=09e7d101148640afa58a193be8431d7c]) successfully deleted
W20260924 15:21:05.051432 13293 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.12.65:43305 (error 108)
I20260924 15:21:05.051606 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 13507
I20260924 15:21:05.054208 13785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.054049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:43305 (local address 127.0.0.1:52732)
0924 15:21:05.054113 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:05.054168 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.65:43305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260924 15:21:05.054363 13293 catalog_manager.cc:4584] TS f9b26414cd9249ea8edceab73391e11e (127.31.12.65:43305): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc84ff9bce5d476ca5bf9daf6c62afee: Network error: Client connection negotiation failed: client connection to 127.31.12.65:43305: connect: Connection refused (error 111)
I20260924 15:21:05.057791 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 13644
I20260924 15:21:05.062222 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 13274
I20260924 15:21:05.067130 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:33699
--webserver_interface=127.31.12.126
--webserver_port=38853
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:33699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:21:05.147550 13818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:05.147780 13818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:05.147819 13818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:05.148980 13818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:21:05.149055 13818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:05.149081 13818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:05.149111 13818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:21:05.149134 13818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:21:05.150641 13818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:33699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:33699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=38853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:05.150923 13818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:05.151144 13818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:05.161778 13824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:05.161770 13827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:05.161777 13825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:05.162077 13818 server_base.cc:1034] running on GCE node
I20260924 15:21:05.162345 13818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:05.162683 13818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:05.163851 13818 hybrid_clock.cc:648] HybridClock initialized: now 1790263265163847 us; error 36 us; skew 500 ppm
Sep 24 15:21:05 dist-test-slave-zwbb krb5kdc[13261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263265, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:05.171336 13818 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:21:05.171779 13818 webserver.cc:460] Webserver started at http://127.31.12.126:38853/ using document root <none> and password file <none>
I20260924 15:21:05.172055 13818 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:05.172139 13818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:05.173535 13818 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:21:05.174196 13834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:05.174358 13818 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:21:05.174464 13818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "063f4028532a4ffbb3a633080866adbb"
format_stamp: "Formatted at 2026-09-24 15:21:04 on dist-test-slave-zwbb"
server_key: "8b7a2169c8d579cb00d442f2cafbb94a"
server_key_iv: "4b631f4e0dd583f40b452a05fe97e1b9"
server_key_version: "encryptionkey@0"
I20260924 15:21:05.174866 13818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:05.191795 13818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:05.194118 13818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:05.194279 13818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:05.198284 13818 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:33699
I20260924 15:21:05.198323 13886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:33699 every 8 connection(s)
I20260924 15:21:05.198732 13818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:21:05.200335 13887 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:21:05.201012 13887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb: Bootstrap starting.
I20260924 15:21:05.201889 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 13818
I20260924 15:21:05.202168 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:43305
--local_ip_for_outbound_sockets=127.31.12.65
--tserver_master_addrs=127.31.12.126:33699
--webserver_port=41225
--webserver_interface=127.31.12.65
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:21:05.210223 13887 log.cc:824] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:05.212170 13887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:21:05.212481 13887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb: Bootstrap complete.
I20260924 15:21:05.214937 13887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f4028532a4ffbb3a633080866adbb" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33699 } }
I20260924 15:21:05.215193 13887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063f4028532a4ffbb3a633080866adbb, State: Initialized, Role: FOLLOWER
I20260924 15:21:05.215399 13887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f4028532a4ffbb3a633080866adbb" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33699 } }
I20260924 15:21:05.215497 13887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:21:05.215559 13887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:21:05.215605 13887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 1 FOLLOWER]: Advancing to term 2
I20260924 15:21:05.216665 13887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f4028532a4ffbb3a633080866adbb" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33699 } }
I20260924 15:21:05.216786 13887 leader_election.cc:302] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 063f4028532a4ffbb3a633080866adbb; no voters:
I20260924 15:21:05.216984 13887 leader_election.cc:288] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 15:21:05.217020 13891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 2 FOLLOWER]: Leader election won for term 2
I20260924 15:21:05.217132 13891 raft_consensus.cc:686] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 2 LEADER]: Becoming Leader. State: Replica: 063f4028532a4ffbb3a633080866adbb, State: Running, Role: LEADER
I20260924 15:21:05.217267 13887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:21:05.217299 13891 consensus_queue.cc:237] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f4028532a4ffbb3a633080866adbb" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33699 } }
I20260924 15:21:05.217689 13892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "063f4028532a4ffbb3a633080866adbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f4028532a4ffbb3a633080866adbb" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33699 } } }
I20260924 15:21:05.217705 13893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 063f4028532a4ffbb3a633080866adbb. Latest consensus state: current_term: 2 leader_uuid: "063f4028532a4ffbb3a633080866adbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f4028532a4ffbb3a633080866adbb" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33699 } } }
I20260924 15:21:05.217826 13893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [sys.catalog]: This master's current role is: LEADER
I20260924 15:21:05.217782 13892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [sys.catalog]: This master's current role is: LEADER
I20260924 15:21:05.218330 13899 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:21:05.219139 13899 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:21:05.219338 13899 catalog_manager.cc:1259] Loaded cluster ID: 52dd219a9bdf4a6a952d443fbed3ee85
I20260924 15:21:05.219564 13899 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:21:05.219846 13899 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 15:21:05.224529 13899 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 15:21:05.227017 13899 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:21:05.227211 13899 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb: Loaded TSK: 0
I20260924 15:21:05.227425 13899 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 15:21:05.284626 13889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:05.284864 13889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:05.284905 13889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:05.286080 13889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:05.286144 13889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:05.286187 13889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:21:05.287675 13889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:43305
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=41225
--tserver_master_addrs=127.31.12.126:33699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:05.287925 13889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:05.288144 13889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:05.292999 13915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:05.293004 13914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:05.293233 13917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:05.293344 13889 server_base.cc:1034] running on GCE node
I20260924 15:21:05.293668 13889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:05.293924 13889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:05.295086 13889 hybrid_clock.cc:648] HybridClock initialized: now 1790263265295074 us; error 30 us; skew 500 ppm
Sep 24 15:21:05 dist-test-slave-zwbb krb5kdc[13261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263265, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:05.297638 13889 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:21:05.298123 13889 webserver.cc:460] Webserver started at http://127.31.12.65:41225/ using document root <none> and password file <none>
I20260924 15:21:05.298357 13889 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:05.298408 13889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:05.299826 13889 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:21:05.300757 13924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:05.300999 13889 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:21:05.301191 13889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "f9b26414cd9249ea8edceab73391e11e"
format_stamp: "Formatted at 2026-09-24 15:21:04 on dist-test-slave-zwbb"
server_key: "e4ac8195a6d6b85586c10e2dadfa66f0"
server_key_iv: "0f4086d05f8b305b891be96dcca01eb3"
server_key_version: "encryptionkey@0"
I20260924 15:21:05.301728 13889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:05.344938 13889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:05.347738 13889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:05.347918 13889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:05.348235 13889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:21:05.348816 13931 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 15:21:05.350350 13889 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 15:21:05.350409 13889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 15:21:05.350442 13889 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 15:21:05.351243 13889 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 15:21:05.351284 13889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 15:21:05.351374 13931 tablet_bootstrap.cc:492] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e: Bootstrap starting.
I20260924 15:21:05.358295 13889 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:43305
I20260924 15:21:05.358618 14040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:43305 every 8 connection(s)
I20260924 15:21:05.358717 13889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:21:05.362349 13931 log.cc:824] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:05.364352 13931 tablet_bootstrap.cc:492] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:21:05.364646 13931 tablet_bootstrap.cc:492] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e: Bootstrap complete.
I20260924 15:21:05.366081 13931 ts_tablet_manager.cc:1397] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
Sep 24 15:21:05 dist-test-slave-zwbb krb5kdc[13261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263265, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:21:05.367130 13931 raft_consensus.cc:348] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } } peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } }
I20260924 15:21:05.367453 13931 raft_consensus.cc:729] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b26414cd9249ea8edceab73391e11e, State: Initialized, Role: FOLLOWER
I20260924 15:21:05.367596 13931 consensus_queue.cc:260] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } } peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } }
I20260924 15:21:05.367901 13931 ts_tablet_manager.cc:1428] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260924 15:21:05.368085 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 13889
I20260924 15:21:05.368353 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:33433
--local_ip_for_outbound_sockets=127.31.12.66
--tserver_master_addrs=127.31.12.126:33699
--webserver_port=44903
--webserver_interface=127.31.12.66
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:21:05.370308 14043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.359446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.65:42745)
0924 15:21:05.359657 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:05.359705 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:05.360281 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:05.361279 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:05.361283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:05.361647 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:05.361829 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:05.361834 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.362995 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.362996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.363777 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:05.363779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.364388 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.364390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.364536 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.365028 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:05.365037 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:05.366638 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:05.368710 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:05.368717 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:05.368719 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:05.368931 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:05.369183 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:05.369184 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:05.369184 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:05.369292 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:05.369710 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:05.369717 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:05.369848 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 15:21:05.369952 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 15:21:05.370127 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":117,"threads_started":1}
I20260924 15:21:05.370308 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.359573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:42745 (local address 127.31.12.126:33699)
0924 15:21:05.360892 (+ 1319us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:05.360900 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:05.360937 (+ 37us) server_negotiation.cc:407] Connection header received
0924 15:21:05.361052 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:05.361056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:05.361097 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:05.361247 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:05.361960 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.362872 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.363876 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.364262 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.364393 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.366743 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:05.366747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:05.366748 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:05.366761 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:05.368621 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:05.369023 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:05.369026 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:05.369026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:05.369114 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:05.369360 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:05.369361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:05.369362 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:05.369637 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:05.369730 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:05.370006 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 15:21:05.370126 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1280,"thread_start_us":72,"threads_started":1}
I20260924 15:21:05.371114 14041 heartbeater.cc:347] Connected to a master server at 127.31.12.126:33699
I20260924 15:21:05.371212 14041 heartbeater.cc:464] Registering TS with master...
I20260924 15:21:05.371392 14041 heartbeater.cc:511] Master 127.31.12.126:33699 requested a full tablet report, sending...
I20260924 15:21:05.372136 13851 ts_manager.cc:194] Registered new tserver with Master: f9b26414cd9249ea8edceab73391e11e (127.31.12.65:43305)
I20260924 15:21:05.373214 13851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:42745
I20260924 15:21:05.374154 14041 heartbeater.cc:503] Master 127.31.12.126:33699 was elected leader, sending a full tablet report...
Sep 24 15:21:05 dist-test-slave-zwbb krb5kdc[13261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263265, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for kudu/127.31.12.65@KRBTEST.COM
I20260924 15:21:05.381810 14049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.373018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.31.12.65:43305)
0924 15:21:05.373169 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:05.373175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:05.373186 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:21:05.373240 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:05.373242 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:05.373258 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:05.373337 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:05.374005 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.374899 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.376127 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.376396 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.376448 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.378945 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:05.378953 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:05.378954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:05.378960 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:05.380532 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:05.380991 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:05.380994 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:05.380994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:05.381056 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:05.381254 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:05.381254 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:05.381254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:05.381393 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:05.381509 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:05.381665 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 15:21:05.381743 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260924 15:21:05.381872 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.372778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:43305 (local address 127.0.0.1:52740)
0924 15:21:05.372955 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:05.372990 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:05.373059 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:05.373406 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:05.373412 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:05.373835 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:05.373914 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:05.373917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.374976 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.374977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.376042 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:05.376046 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.376474 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.376475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.376570 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.377044 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:05.377050 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:05.378844 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:05.380617 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:05.380625 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:05.380627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:05.380905 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:05.381122 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:05.381123 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:05.381124 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:05.381194 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:05.381482 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:05.381489 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:05.381545 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 15:21:05.381639 (+ 94us) client_negotiation.cc:240] Negotiation successful
0924 15:21:05.381767 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260924 15:21:05.382366 13975 tablet_service.cc:1514] Processing DeleteTablet for tablet cc84ff9bce5d476ca5bf9daf6c62afee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:21:05 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:52740
I20260924 15:21:05.382936 14050 tablet_replica.cc:330] stopping tablet replica
I20260924 15:21:05.383051 14050 raft_consensus.cc:2229] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:21:05.383121 14050 raft_consensus.cc:2258] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:21:05.383451 14050 ts_tablet_manager.cc:1905] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:21:05.385932 14050 ts_tablet_manager.cc:1918] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:21:05.386018 14050 log.cc:1196] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/wal/wals/cc84ff9bce5d476ca5bf9daf6c62afee
I20260924 15:21:05.386250 14050 ts_tablet_manager.cc:1939] T cc84ff9bce5d476ca5bf9daf6c62afee P f9b26414cd9249ea8edceab73391e11e: Deleting consensus metadata
I20260924 15:21:05.386787 13838 catalog_manager.cc:4841] TS f9b26414cd9249ea8edceab73391e11e (127.31.12.65:43305): tablet cc84ff9bce5d476ca5bf9daf6c62afee (table test-table [id=09e7d101148640afa58a193be8431d7c]) successfully deleted
W20260924 15:21:05.454998 14047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:05.455192 14047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:05.455216 14047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:05.456295 14047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:05.456331 14047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:05.456357 14047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:21:05.457741 14047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:33433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=44903
--tserver_master_addrs=127.31.12.126:33699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:05.457916 14047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:05.458083 14047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:05.463266 14055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:05.463277 14056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:05.463433 14047 server_base.cc:1034] running on GCE node
W20260924 15:21:05.463282 14058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:05.463745 14047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:05.463994 14047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:05.465137 14047 hybrid_clock.cc:648] HybridClock initialized: now 1790263265465124 us; error 27 us; skew 500 ppm
Sep 24 15:21:05 dist-test-slave-zwbb krb5kdc[13261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263265, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:05.467522 14047 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:21:05.467993 14047 webserver.cc:460] Webserver started at http://127.31.12.66:44903/ using document root <none> and password file <none>
I20260924 15:21:05.468214 14047 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:05.468286 14047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:05.469652 14047 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:21:05.470248 14065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:05.470433 14047 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:21:05.470526 14047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "23c27fce05f440d99056a2db20681a73"
format_stamp: "Formatted at 2026-09-24 15:21:04 on dist-test-slave-zwbb"
server_key: "18b601ae31e2afc2ff83b4390f2ed1d1"
server_key_iv: "66e56b913285f6720ef56593eadc4105"
server_key_version: "encryptionkey@0"
I20260924 15:21:05.470855 14047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:05.484802 14047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:05.486629 14047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:05.486784 14047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:05.487015 14047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:21:05.487308 14047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:21:05.487341 14047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:05.487361 14047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:21:05.487406 14047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:05.493237 14047 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:33433
I20260924 15:21:05.493273 14179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:33433 every 8 connection(s)
I20260924 15:21:05.493669 14047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:21:05.493927 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 14047
I20260924 15:21:05.494302 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:41053
--local_ip_for_outbound_sockets=127.31.12.67
--tserver_master_addrs=127.31.12.126:33699
--webserver_port=39879
--webserver_interface=127.31.12.67
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 15:21:05 dist-test-slave-zwbb krb5kdc[13261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263265, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:21:05.504390 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.494377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.66:54489)
0924 15:21:05.494717 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:05.494767 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:05.495564 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:05.495819 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:05.495823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:05.496668 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:05.496947 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:05.496953 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.497854 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.497855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.498664 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:05.498667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.499256 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.499257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.499356 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.499855 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:05.499862 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:05.501180 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:05.503086 (+ 1906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:05.503092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:05.503094 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:05.503305 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:05.503540 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:05.503540 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:05.503541 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:05.503633 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:05.503929 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:05.503933 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:05.504042 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 15:21:05.504123 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 15:21:05.504236 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":105,"threads_started":1}
I20260924 15:21:05.504426 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.494392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:54489 (local address 127.31.12.126:33699)
0924 15:21:05.494448 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:05.494450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:05.494873 (+ 423us) server_negotiation.cc:407] Connection header received
0924 15:21:05.495667 (+ 794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:05.495669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:05.495684 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:05.495739 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:05.497045 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.497766 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.498769 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.499160 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.499237 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.501277 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:05.501279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:05.501279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:05.501289 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:05.503004 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:05.503382 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:05.503384 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:05.503384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:05.503462 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:05.503699 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:05.503700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:05.503700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:05.503859 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:05.503938 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:05.504174 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 15:21:05.504277 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 15:21:05.504948 14180 heartbeater.cc:347] Connected to a master server at 127.31.12.126:33699
I20260924 15:21:05.505059 14180 heartbeater.cc:464] Registering TS with master...
I20260924 15:21:05.505327 14180 heartbeater.cc:511] Master 127.31.12.126:33699 requested a full tablet report, sending...
I20260924 15:21:05.505774 13851 ts_manager.cc:194] Registered new tserver with Master: 23c27fce05f440d99056a2db20681a73 (127.31.12.66:33433)
I20260924 15:21:05.506214 13851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:54489
W20260924 15:21:05.579423 14183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:05.579636 14183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:05.579659 14183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:05.580818 14183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:05.580859 14183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:05.580886 14183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:21:05.582355 14183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:41053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=39879
--tserver_master_addrs=127.31.12.126:33699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:05.582605 14183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:05.582845 14183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:05.587926 14188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:05.587921 14189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:05.587947 14183 server_base.cc:1034] running on GCE node
W20260924 15:21:05.587921 14191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:05.588346 14183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:05.588554 14183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:05.589704 14183 hybrid_clock.cc:648] HybridClock initialized: now 1790263265589683 us; error 37 us; skew 500 ppm
Sep 24 15:21:05 dist-test-slave-zwbb krb5kdc[13261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263265, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:05.592239 14183 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:21:05.592653 14183 webserver.cc:460] Webserver started at http://127.31.12.67:39879/ using document root <none> and password file <none>
I20260924 15:21:05.592891 14183 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:05.592972 14183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:05.594358 14183 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:21:05.594918 14198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:05.595113 14183 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:21:05.595201 14183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "825a0a4a54e14c3dbcda29a6354c51b0"
format_stamp: "Formatted at 2026-09-24 15:21:04 on dist-test-slave-zwbb"
server_key: "a250e0d320f3a06e03e0309cb0ebe3ee"
server_key_iv: "6d5f6ab1f945218046111fcd2123bc9e"
server_key_version: "encryptionkey@0"
I20260924 15:21:05.595515 14183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:05.612165 14183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:05.614454 14183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:05.614574 14183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:05.614773 14183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:21:05.615175 14183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:21:05.615234 14183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:05.615270 14183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:21:05.615293 14183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:05.621798 14183 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:41053
I20260924 15:21:05.621883 14311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:41053 every 8 connection(s)
I20260924 15:21:05.622258 14183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:21:05 dist-test-slave-zwbb krb5kdc[13261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263265, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:21:05.630505 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 14183
I20260924 15:21:05.631637 14044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.623046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:46067 (local address 127.31.12.126:33699)
0924 15:21:05.623121 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:05.623123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:05.623304 (+ 181us) server_negotiation.cc:407] Connection header received
0924 15:21:05.623812 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:05.623813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:05.623824 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:05.623890 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:05.624840 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.625352 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.626373 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.626565 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.626623 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.628766 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:05.628768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:05.628768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:05.628774 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:05.630054 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:05.630431 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:05.630432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:05.630433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:05.630503 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:05.630842 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:05.630843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:05.630843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:05.631017 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:05.631094 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:05.631442 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 15:21:05.631528 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:21:05.631659 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.623009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.67:46067)
0924 15:21:05.623167 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:05.623209 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:05.623697 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:05.623970 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:05.623974 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:05.624558 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:05.624752 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:05.624757 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.625431 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.625432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.626295 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:05.626297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.626640 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.626641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.626743 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.627286 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:05.627295 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:05.628682 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:05.630128 (+ 1446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:05.630134 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:05.630135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:05.630353 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:05.630593 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:05.630594 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:05.630594 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:05.630754 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:05.631100 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:05.631108 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:05.631279 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 15:21:05.631376 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 15:21:05.631500 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
I20260924 15:21:05.632136 14312 heartbeater.cc:347] Connected to a master server at 127.31.12.126:33699
I20260924 15:21:05.632261 14312 heartbeater.cc:464] Registering TS with master...
I20260924 15:21:05.632438 14312 heartbeater.cc:511] Master 127.31.12.126:33699 requested a full tablet report, sending...
I20260924 15:21:05.632898 13851 ts_manager.cc:194] Registered new tserver with Master: 825a0a4a54e14c3dbcda29a6354c51b0 (127.31.12.67:41053)
I20260924 15:21:05.633265 13851 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:46067
I20260924 15:21:05.637233 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.630919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45770 (local address 127.31.12.126:33699)
0924 15:21:05.631033 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:05.631035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:05.631095 (+ 60us) server_negotiation.cc:407] Connection header received
0924 15:21:05.631149 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:05.631150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:05.631162 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:05.631229 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:05.632005 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.632695 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.633588 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.633815 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.633869 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.634982 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:05.634983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:05.634984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:05.634989 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:05.636344 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:05.636638 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:05.636639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:05.636639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:05.636689 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:05.636849 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:05.636850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:05.636850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:05.636936 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:05.637055 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:05.637087 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 15:21:05.637145 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":47,"threads_started":1}
I20260924 15:21:05.638001 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:21:05.640803 14316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.638336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45778 (local address 127.31.12.126:33699)
0924 15:21:05.638399 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:05.638401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:05.638499 (+ 98us) server_negotiation.cc:407] Connection header received
0924 15:21:05.638595 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:05.638596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:05.638631 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:05.638709 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:21:05.638842 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.639292 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.640029 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.640239 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.640313 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.640487 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:21:05.640589 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:21:05.640671 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:05.640711 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 15:21:05.640732 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 15:21:05.641083 13851 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:21:05.642771 13851 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:21:05.645021 13975 tablet_service.cc:1467] Processing CreateTablet for tablet 48f4d274a359415382718dd00609e19b (DEFAULT_TABLE table=test-table [id=30fcd6241fe7452f911295718960db7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:21:05.645226 13975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f4d274a359415382718dd00609e19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:21:05.646581 13931 tablet_bootstrap.cc:492] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e: Bootstrap starting.
I20260924 15:21:05.646924 13931 tablet_bootstrap.cc:654] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e: Neither blocks nor log segments found. Creating new log.
I20260924 15:21:05.647462 13931 tablet_bootstrap.cc:492] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e: No bootstrap required, opened a new log
I20260924 15:21:05.647516 13931 ts_tablet_manager.cc:1397] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 15:21:05.647877 13931 raft_consensus.cc:348] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } } peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } }
I20260924 15:21:05.647964 13931 raft_consensus.cc:374] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:21:05.648022 13931 raft_consensus.cc:729] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b26414cd9249ea8edceab73391e11e, State: Initialized, Role: FOLLOWER
I20260924 15:21:05.648068 13931 consensus_queue.cc:260] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } } peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } }
I20260924 15:21:05.648165 13931 ts_tablet_manager.cc:1428] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 15:21:05.649115 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.644900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60788 (local address 127.31.12.66:33433)
0924 15:21:05.645241 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:05.645246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:05.645256 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:21:05.645293 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:05.645295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:05.645311 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:05.645418 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:05.645647 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.646720 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.647995 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.648761 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.648831 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.648890 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:05.648996 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 15:21:05.649033 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":122,"threads_started":1}
I20260924 15:21:05.649140 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.644565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35816 (local address 127.31.12.67:41053)
0924 15:21:05.644900 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:05.644904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:05.644913 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:21:05.644943 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:05.644945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:05.644957 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:05.645048 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:05.645208 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.646412 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.647837 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.648760 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.648860 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.648919 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:05.649016 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 15:21:05.649059 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":103,"threads_started":1}
I20260924 15:21:05.649134 14048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.644458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:41053 (local address 127.0.0.1:35816)
0924 15:21:05.644508 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:05.644519 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:05.644565 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:05.645061 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:05.645062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:05.645064 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:05.645106 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:05.645107 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.646526 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.646527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.647748 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:05.647750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.648863 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.648865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.648942 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.648946 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:21:05.649003 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:21:05.649051 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 15:21:05.649135 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.644565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:33433 (local address 127.0.0.1:60788)
0924 15:21:05.644714 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:05.644723 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:05.644763 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:05.645508 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:05.645510 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:05.645512 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:05.645549 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:05.645550 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.646825 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.646826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.647913 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:05.647914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.648831 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.648832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.648895 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.648916 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 15:21:05.648974 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 15:21:05.649017 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":111,"threads_started":1}
I20260924 15:21:05.649833 14246 tablet_service.cc:1467] Processing CreateTablet for tablet 48f4d274a359415382718dd00609e19b (DEFAULT_TABLE table=test-table [id=30fcd6241fe7452f911295718960db7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:21:05.649979 14113 tablet_service.cc:1467] Processing CreateTablet for tablet 48f4d274a359415382718dd00609e19b (DEFAULT_TABLE table=test-table [id=30fcd6241fe7452f911295718960db7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:21:05.650192 14246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f4d274a359415382718dd00609e19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:21:05.650300 14113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48f4d274a359415382718dd00609e19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:21:05.653738 14323 tablet_bootstrap.cc:492] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0: Bootstrap starting.
I20260924 15:21:05.654335 14323 tablet_bootstrap.cc:654] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0: Neither blocks nor log segments found. Creating new log.
I20260924 15:21:05.654824 14323 log.cc:824] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:05.655128 14322 tablet_bootstrap.cc:492] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73: Bootstrap starting.
I20260924 15:21:05.655606 14323 tablet_bootstrap.cc:492] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0: No bootstrap required, opened a new log
I20260924 15:21:05.655678 14323 ts_tablet_manager.cc:1397] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:21:05.655766 14322 tablet_bootstrap.cc:654] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73: Neither blocks nor log segments found. Creating new log.
I20260924 15:21:05.656307 14322 log.cc:824] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:05.657136 14322 tablet_bootstrap.cc:492] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73: No bootstrap required, opened a new log
I20260924 15:21:05.657214 14322 ts_tablet_manager.cc:1397] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:21:05.657676 14323 raft_consensus.cc:348] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } } peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } }
I20260924 15:21:05.657799 14323 raft_consensus.cc:374] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:21:05.657819 14323 raft_consensus.cc:729] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 825a0a4a54e14c3dbcda29a6354c51b0, State: Initialized, Role: FOLLOWER
I20260924 15:21:05.657943 14323 consensus_queue.cc:260] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } } peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } }
I20260924 15:21:05.658215 14323 ts_tablet_manager.cc:1428] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 15:21:05.658229 14312 heartbeater.cc:503] Master 127.31.12.126:33699 was elected leader, sending a full tablet report...
I20260924 15:21:05.658998 14322 raft_consensus.cc:348] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } } peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } }
I20260924 15:21:05.659119 14322 raft_consensus.cc:374] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:21:05.659140 14322 raft_consensus.cc:729] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23c27fce05f440d99056a2db20681a73, State: Initialized, Role: FOLLOWER
I20260924 15:21:05.659260 14322 consensus_queue.cc:260] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } } peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } }
I20260924 15:21:05.659539 14322 ts_tablet_manager.cc:1428] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 15:21:05.659562 14180 heartbeater.cc:503] Master 127.31.12.126:33699 was elected leader, sending a full tablet report...
W20260924 15:21:05.744320 14181 tablet.cc:2369] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:21:05.861166 14042 tablet.cc:2369] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:21:05.872877 14313 tablet.cc:2369] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:21:05.970425 14327 raft_consensus.cc:482] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:21:05.970590 14327 raft_consensus.cc:504] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } } peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } }
I20260924 15:21:05.971063 14327 leader_election.cc:288] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 825a0a4a54e14c3dbcda29a6354c51b0 (127.31.12.67:41053), f9b26414cd9249ea8edceab73391e11e (127.31.12.65:43305)
I20260924 15:21:05.975440 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.971494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:47955 (local address 127.31.12.65:43305)
0924 15:21:05.971615 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:05.971617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:05.971625 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:21:05.971661 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:05.971661 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:05.971685 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:05.971777 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:05.971933 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.972650 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.974316 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.975027 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.975101 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.975191 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:05.975308 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 15:21:05.975355 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260924 15:21:05.975440 14318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.971382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:55209 (local address 127.31.12.67:41053)
0924 15:21:05.971452 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:05.971455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:05.971466 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:21:05.971499 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:05.971500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:05.971510 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:05.971598 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:05.971783 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.972551 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.974316 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.975000 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.975096 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.975106 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:05.975308 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 15:21:05.975357 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 15:21:05.975446 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.971282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:41053 (local address 127.31.12.66:55209)
0924 15:21:05.971345 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:05.971358 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:05.971410 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:05.971620 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:05.971620 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:05.971622 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:05.971704 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:05.971706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.972657 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.972659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.974148 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:05.974152 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.975116 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.975117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.975216 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.975218 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:21:05.975284 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 15:21:05.975340 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":16}
I20260924 15:21:05.975456 14328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.971424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:43305 (local address 127.31.12.66:47955)
0924 15:21:05.971524 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:05.971533 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:05.971571 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:05.971782 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:05.971783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:05.971785 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:05.971854 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:05.971855 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.972742 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.972742 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.974156 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:05.974157 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.975116 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.975117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.975210 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.975215 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:21:05.975285 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 15:21:05.975339 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":30,"thread_start_us":61,"threads_started":1}
I20260924 15:21:05.975859 14266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f4d274a359415382718dd00609e19b" candidate_uuid: "23c27fce05f440d99056a2db20681a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" is_pre_election: true
I20260924 15:21:05.975823 13995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f4d274a359415382718dd00609e19b" candidate_uuid: "23c27fce05f440d99056a2db20681a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b26414cd9249ea8edceab73391e11e" is_pre_election: true
I20260924 15:21:05.976035 14266 raft_consensus.cc:2454] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23c27fce05f440d99056a2db20681a73 in term 0.
I20260924 15:21:05.976035 13995 raft_consensus.cc:2454] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23c27fce05f440d99056a2db20681a73 in term 0.
I20260924 15:21:05.976105 14326 raft_consensus.cc:482] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:21:05.976178 14326 raft_consensus.cc:504] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } } peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } }
I20260924 15:21:05.976297 14069 leader_election.cc:302] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23c27fce05f440d99056a2db20681a73, f9b26414cd9249ea8edceab73391e11e; no voters:
I20260924 15:21:05.976557 14327 raft_consensus.cc:2781] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:21:05.976588 14326 leader_election.cc:288] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23c27fce05f440d99056a2db20681a73 (127.31.12.66:33433), f9b26414cd9249ea8edceab73391e11e (127.31.12.65:43305)
I20260924 15:21:05.976634 14327 raft_consensus.cc:482] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:21:05.976670 14327 raft_consensus.cc:3037] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:21:05.977792 14327 raft_consensus.cc:504] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } } peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } }
I20260924 15:21:05.977957 14327 leader_election.cc:288] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [CANDIDATE]: Term 1 election: Requested vote from peers 825a0a4a54e14c3dbcda29a6354c51b0 (127.31.12.67:41053), f9b26414cd9249ea8edceab73391e11e (127.31.12.65:43305)
I20260924 15:21:05.978168 13995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f4d274a359415382718dd00609e19b" candidate_uuid: "23c27fce05f440d99056a2db20681a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b26414cd9249ea8edceab73391e11e"
I20260924 15:21:05.978195 14266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f4d274a359415382718dd00609e19b" candidate_uuid: "23c27fce05f440d99056a2db20681a73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "825a0a4a54e14c3dbcda29a6354c51b0"
I20260924 15:21:05.978267 13995 raft_consensus.cc:3037] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:21:05.978282 14266 raft_consensus.cc:3037] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:21:05.979188 13995 raft_consensus.cc:2454] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23c27fce05f440d99056a2db20681a73 in term 1.
I20260924 15:21:05.979233 14266 raft_consensus.cc:2454] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23c27fce05f440d99056a2db20681a73 in term 1.
I20260924 15:21:05.979573 14069 leader_election.cc:302] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23c27fce05f440d99056a2db20681a73, 825a0a4a54e14c3dbcda29a6354c51b0; no voters:
I20260924 15:21:05.979705 14327 raft_consensus.cc:2781] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:21:05.979853 14327 raft_consensus.cc:686] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [term 1 LEADER]: Becoming Leader. State: Replica: 23c27fce05f440d99056a2db20681a73, State: Running, Role: LEADER
I20260924 15:21:05.979969 14327 consensus_queue.cc:237] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } } peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } }
I20260924 15:21:05.980613 14314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.976762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:33433 (local address 127.31.12.67:57291)
0924 15:21:05.976803 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:05.976816 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:05.976870 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:05.977292 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:05.977293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:05.977295 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:05.977359 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:05.977361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.978229 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.978230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.979519 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:05.979522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.980313 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.980314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.980409 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.980415 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 15:21:05.980496 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 15:21:05.980540 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 15:21:05.980612 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.977096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:57291 (local address 127.31.12.66:33433)
0924 15:21:05.977156 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:05.977158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:05.977165 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:21:05.977188 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:05.977188 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:05.977197 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:05.977253 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:05.977497 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.978124 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.979620 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.980231 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.980311 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.980320 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:05.980510 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 15:21:05.980548 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:21:05.980976 14133 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f4d274a359415382718dd00609e19b" candidate_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23c27fce05f440d99056a2db20681a73" is_pre_election: true
I20260924 15:21:05.981703 14330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.976915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:43305 (local address 127.31.12.67:53295)
0924 15:21:05.978071 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:05.978083 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:05.978141 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:05.978328 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:05.978329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:05.978331 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:05.978470 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:05.978473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.979382 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.979383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.980602 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:05.980605 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.981412 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.981413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:05.981531 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.981535 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:21:05.981600 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 15:21:05.981646 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1135,"mutex_wait_us":63,"thread_start_us":87,"threads_started":1}
I20260924 15:21:05.981722 14329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.976979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:53295 (local address 127.31.12.65:43305)
0924 15:21:05.977018 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:05.977020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:05.978200 (+ 1180us) server_negotiation.cc:407] Connection header received
0924 15:21:05.978223 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:05.978224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:05.978235 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:05.978283 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:05.978566 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.979276 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.980705 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.981280 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:05.981353 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:05.981363 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:05.981625 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 15:21:05.981666 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:21:05.981743 13851 catalog_manager.cc:5495] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23c27fce05f440d99056a2db20681a73 (127.31.12.66). New cstate: current_term: 1 leader_uuid: "23c27fce05f440d99056a2db20681a73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } health_report { overall_health: UNKNOWN } } }
I20260924 15:21:05.982254 13995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48f4d274a359415382718dd00609e19b" candidate_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9b26414cd9249ea8edceab73391e11e" is_pre_election: true
I20260924 15:21:05.982352 13995 raft_consensus.cc:2379] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 825a0a4a54e14c3dbcda29a6354c51b0 in current term 1: Already voted for candidate 23c27fce05f440d99056a2db20681a73 in this term.
I20260924 15:21:05.982578 14202 leader_election.cc:302] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 825a0a4a54e14c3dbcda29a6354c51b0; no voters: 23c27fce05f440d99056a2db20681a73, f9b26414cd9249ea8edceab73391e11e
I20260924 15:21:05.982775 14326 raft_consensus.cc:2726] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 15:21:06.001245 14320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:05.998796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60798 (local address 127.31.12.66:33433)
0924 15:21:05.998864 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:05.998867 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:05.998929 (+ 62us) server_negotiation.cc:407] Connection header received
0924 15:21:05.998975 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:05.998976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:05.998994 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:05.999058 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:21:05.999252 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:05.999752 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:06.000466 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:06.000673 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:06.000737 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:06.000892 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:21:06.001000 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:21:06.001068 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:06.001087 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 15:21:06.001104 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 15:21:06.003075 13995 raft_consensus.cc:1261] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Refusing update from remote peer 23c27fce05f440d99056a2db20681a73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:21:06.003098 14266 raft_consensus.cc:1261] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0 [term 1 FOLLOWER]: Refusing update from remote peer 23c27fce05f440d99056a2db20681a73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:21:06.003439 14327 consensus_queue.cc:1035] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:21:06.003613 14331 consensus_queue.cc:1035] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:21:06.006259 14336 mvcc.cc:205] Tried to move back new op lower bound from 7332918337545891840 to 7332918337454583808. Current Snapshot: MvccSnapshot[applied={T|T < 7332918337545891840}]
I20260924 15:21:06.006716 14335 mvcc.cc:205] Tried to move back new op lower bound from 7332918337545891840 to 7332918337454583808. Current Snapshot: MvccSnapshot[applied={T|T < 7332918337545891840}]
I20260924 15:21:06.008430 13851 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:21:06.008541 13851 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:21:06.009351 13851 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb: Sending DeleteTablet for 3 replicas of tablet 48f4d274a359415382718dd00609e19b
I20260924 15:21:06.009693 14246 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f4d274a359415382718dd00609e19b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:21:06 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:35816
I20260924 15:21:06.009696 14113 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f4d274a359415382718dd00609e19b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:21:06 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:60788
I20260924 15:21:06.009743 13975 tablet_service.cc:1514] Processing DeleteTablet for tablet 48f4d274a359415382718dd00609e19b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:21:06 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:52740
I20260924 15:21:06.009985 14345 tablet_replica.cc:330] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e: stopping tablet replica
I20260924 15:21:06.010010 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 13889
I20260924 15:21:06.010047 14345 raft_consensus.cc:2229] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:21:06.010102 14345 raft_consensus.cc:2258] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:21:06.012689 14346 tablet_replica.cc:330] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73: stopping tablet replica
I20260924 15:21:06.012773 14346 raft_consensus.cc:2229] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:21:06.012914 14346 raft_consensus.cc:2258] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:21:06.013412 14346 ts_tablet_manager.cc:1905] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:21:06.015466 14346 ts_tablet_manager.cc:1918] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:21:06.015554 14346 log.cc:1196] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/48f4d274a359415382718dd00609e19b
I20260924 15:21:06.015852 14346 ts_tablet_manager.cc:1939] T 48f4d274a359415382718dd00609e19b P 23c27fce05f440d99056a2db20681a73: Deleting consensus metadata
I20260924 15:21:06.016289 13838 catalog_manager.cc:4841] TS 23c27fce05f440d99056a2db20681a73 (127.31.12.66:33433): tablet 48f4d274a359415382718dd00609e19b (table test-table [id=30fcd6241fe7452f911295718960db7c]) successfully deleted
W20260924 15:21:06.016510 13838 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.12.65:43305 (error 108)
I20260924 15:21:06.016776 14344 tablet_replica.cc:330] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0: stopping tablet replica
I20260924 15:21:06.016906 14344 raft_consensus.cc:2229] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:21:06.017019 14344 raft_consensus.cc:2258] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 15:21:06.017380 14319 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:21:06.016852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:43305 (local address 127.0.0.1:52754)
0924 15:21:06.016920 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:06.016932 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:06.016979 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:06.017329 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.65:43305: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 15:21:06.017583 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 14047
W20260924 15:21:06.017573 13838 catalog_manager.cc:4584] TS f9b26414cd9249ea8edceab73391e11e (127.31.12.65:43305): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48f4d274a359415382718dd00609e19b: Network error: Client connection negotiation failed: client connection to 127.31.12.65:43305: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 15:21:06.019618 14344 ts_tablet_manager.cc:1905] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:21:06.020967 14344 ts_tablet_manager.cc:1918] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:21:06.021060 14344 log.cc:1196] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/48f4d274a359415382718dd00609e19b
I20260924 15:21:06.021628 14344 ts_tablet_manager.cc:1939] T 48f4d274a359415382718dd00609e19b P 825a0a4a54e14c3dbcda29a6354c51b0: Deleting consensus metadata
I20260924 15:21:06.022025 13838 catalog_manager.cc:4841] TS 825a0a4a54e14c3dbcda29a6354c51b0 (127.31.12.67:41053): tablet 48f4d274a359415382718dd00609e19b (table test-table [id=30fcd6241fe7452f911295718960db7c]) successfully deleted
I20260924 15:21:06.023900 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 14183
I20260924 15:21:06.029207 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 13818
I20260924 15:21:06.034691 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:33699
--webserver_interface=127.31.12.126
--webserver_port=38853
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:33699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:21:06.114230 14348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:06.114413 14348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:06.114434 14348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:06.115566 14348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:21:06.115607 14348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:06.115620 14348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:06.115659 14348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:21:06.115689 14348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:21:06.117120 14348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:33699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:33699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=38853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:06.117291 14348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:06.117556 14348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:06.122155 14356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:06.122202 14354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:06.122154 14353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:06.122460 14348 server_base.cc:1034] running on GCE node
I20260924 15:21:06.122699 14348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:06.122972 14348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:06.124130 14348 hybrid_clock.cc:648] HybridClock initialized: now 1790263266124118 us; error 28 us; skew 500 ppm
Sep 24 15:21:06 dist-test-slave-zwbb krb5kdc[13261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263266, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:06.126488 14348 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:21:06.126896 14348 webserver.cc:460] Webserver started at http://127.31.12.126:38853/ using document root <none> and password file <none>
I20260924 15:21:06.127130 14348 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:06.127199 14348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:06.128463 14348 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:21:06.129120 14363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:06.129330 14348 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:21:06.129477 14348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "063f4028532a4ffbb3a633080866adbb"
format_stamp: "Formatted at 2026-09-24 15:21:04 on dist-test-slave-zwbb"
server_key: "8b7a2169c8d579cb00d442f2cafbb94a"
server_key_iv: "4b631f4e0dd583f40b452a05fe97e1b9"
server_key_version: "encryptionkey@0"
I20260924 15:21:06.129822 14348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:06.162266 14348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:06.164105 14348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:06.164270 14348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:06.168226 14348 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:33699
I20260924 15:21:06.168275 14416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:33699 every 8 connection(s)
I20260924 15:21:06.168699 14348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:21:06.169871 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 14348
I20260924 15:21:06.170171 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:43305
--local_ip_for_outbound_sockets=127.31.12.65
--tserver_master_addrs=127.31.12.126:33699
--webserver_port=41225
--webserver_interface=127.31.12.65
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:21:06.170662 14417 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:21:06.171779 14417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb: Bootstrap starting.
I20260924 15:21:06.180229 14417 log.cc:824] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:06.182452 14417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:21:06.182754 14417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb: Bootstrap complete.
I20260924 15:21:06.184928 14417 raft_consensus.cc:348] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f4028532a4ffbb3a633080866adbb" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33699 } }
I20260924 15:21:06.185150 14417 raft_consensus.cc:729] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 063f4028532a4ffbb3a633080866adbb, State: Initialized, Role: FOLLOWER
I20260924 15:21:06.185312 14417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f4028532a4ffbb3a633080866adbb" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33699 } }
I20260924 15:21:06.185370 14417 raft_consensus.cc:388] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:21:06.185427 14417 raft_consensus.cc:482] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:21:06.185495 14417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 2 FOLLOWER]: Advancing to term 3
I20260924 15:21:06.186442 14417 raft_consensus.cc:504] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f4028532a4ffbb3a633080866adbb" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33699 } }
I20260924 15:21:06.186528 14417 leader_election.cc:302] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 063f4028532a4ffbb3a633080866adbb; no voters:
I20260924 15:21:06.186735 14417 leader_election.cc:288] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 15:21:06.186767 14422 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 3 FOLLOWER]: Leader election won for term 3
I20260924 15:21:06.186869 14422 raft_consensus.cc:686] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [term 3 LEADER]: Becoming Leader. State: Replica: 063f4028532a4ffbb3a633080866adbb, State: Running, Role: LEADER
I20260924 15:21:06.186961 14422 consensus_queue.cc:237] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f4028532a4ffbb3a633080866adbb" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33699 } }
I20260924 15:21:06.187008 14417 sys_catalog.cc:564] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:21:06.187302 14422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "063f4028532a4ffbb3a633080866adbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f4028532a4ffbb3a633080866adbb" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33699 } } }
I20260924 15:21:06.187376 14422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [sys.catalog]: This master's current role is: LEADER
I20260924 15:21:06.187507 14422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 063f4028532a4ffbb3a633080866adbb. Latest consensus state: current_term: 3 leader_uuid: "063f4028532a4ffbb3a633080866adbb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "063f4028532a4ffbb3a633080866adbb" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33699 } } }
I20260924 15:21:06.187574 14422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 063f4028532a4ffbb3a633080866adbb [sys.catalog]: This master's current role is: LEADER
I20260924 15:21:06.187852 14427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:21:06.188651 14427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:21:06.188910 14427 catalog_manager.cc:1259] Loaded cluster ID: 52dd219a9bdf4a6a952d443fbed3ee85
I20260924 15:21:06.189007 14427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 15:21:06.189229 14427 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f77ab6379ad google::LogMessage::Fail() at ??:0
@ 0x7f77ab63a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f77ab6374de google::LogMessage::Flush() at ??:0
@ 0x7f77ab638a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f77b35580f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f77b35618b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f77ad5050d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f77ad4fa954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f77ae35e6db start_thread at ??:0
@ 0x7f77aa13561f clone at ??:0
W20260924 15:21:06.263592 14419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:06.263815 14419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:06.263854 14419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:06.265074 14419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:06.265133 14419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:06.265177 14419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:21:06.266690 14419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:43305
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=41225
--tserver_master_addrs=127.31.12.126:33699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:06.266963 14419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:06.267195 14419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:06.273132 14455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:06.273233 14419 server_base.cc:1034] running on GCE node
W20260924 15:21:06.273142 14457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:06.273380 14454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:06.273643 14419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:06.273905 14419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:06.275080 14419 hybrid_clock.cc:648] HybridClock initialized: now 1790263266275069 us; error 26 us; skew 500 ppm
Sep 24 15:21:06 dist-test-slave-zwbb krb5kdc[13261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263266, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:06.277736 14419 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:21:06.278146 14419 webserver.cc:460] Webserver started at http://127.31.12.65:41225/ using document root <none> and password file <none>
I20260924 15:21:06.278378 14419 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:06.278450 14419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:06.279814 14419 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:21:06.280552 14464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:06.280728 14419 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:21:06.280828 14419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "f9b26414cd9249ea8edceab73391e11e"
format_stamp: "Formatted at 2026-09-24 15:21:04 on dist-test-slave-zwbb"
server_key: "e4ac8195a6d6b85586c10e2dadfa66f0"
server_key_iv: "0f4086d05f8b305b891be96dcca01eb3"
server_key_version: "encryptionkey@0"
I20260924 15:21:06.281170 14419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:06.298412 14419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:06.300580 14419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:06.300746 14419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:06.300983 14419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:21:06.301501 14471 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 15:21:06.303444 14419 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 15:21:06.303498 14419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 15:21:06.303534 14419 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 15:21:06.304553 14419 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 15:21:06.304597 14419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 15:21:06.304692 14471 tablet_bootstrap.cc:492] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e: Bootstrap starting.
I20260924 15:21:06.312352 14419 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:43305
I20260924 15:21:06.312526 14581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:43305 every 8 connection(s)
I20260924 15:21:06.312862 14419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:21:06.316205 14471 log.cc:824] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:06.316905 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 14419
I20260924 15:21:06.317205 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:33433
--local_ip_for_outbound_sockets=127.31.12.66
--tserver_master_addrs=127.31.12.126:33699
--webserver_port=44903
--webserver_interface=127.31.12.66
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:21:06.319175 14471 tablet_bootstrap.cc:492] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:21:06.319471 14471 tablet_bootstrap.cc:492] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e: Bootstrap complete.
I20260924 15:21:06.320654 14471 ts_tablet_manager.cc:1397] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.007s
I20260924 15:21:06.322463 14471 raft_consensus.cc:348] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } } peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } }
I20260924 15:21:06.322880 14471 raft_consensus.cc:729] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b26414cd9249ea8edceab73391e11e, State: Initialized, Role: FOLLOWER
I20260924 15:21:06.323052 14471 consensus_queue.cc:260] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } } peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } }
I20260924 15:21:06.323355 14471 ts_tablet_manager.cc:1428] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e: Time spent starting tablet: real 0.003s user 0.005s sys 0.001s
W20260924 15:21:06.379736 14584 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:21:06.313644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.65:55973)
0924 15:21:06.313856 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:06.313907 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:06.314562 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:06.379509 (+ 64947us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":105,"threads_started":1}
W20260924 15:21:06.380017 14582 heartbeater.cc:644] Failed to heartbeat to 127.31.12.126:33699 (0 consecutive failures): Network error: Failed to ping master at 127.31.12.126:33699: Client connection negotiation failed: client connection to 127.31.12.126:33699: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 15:21:06.380378 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:06.380250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.65:57103)
0924 15:21:06.380307 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:06.380352 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 15:21:06.380788 14584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:06.380691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.65:42539)
0924 15:21:06.380737 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:06.380763 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 15:21:06.380971 14582 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 15:21:06.400260 14586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:06.400445 14586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:06.400467 14586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:06.401697 14586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:06.401738 14586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:06.401783 14586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:21:06.403178 14586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:33433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=44903
--tserver_master_addrs=127.31.12.126:33699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:06.403405 14586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:06.403630 14586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:06.408684 14596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:06.408720 14586 server_base.cc:1034] running on GCE node
W20260924 15:21:06.408742 14593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:06.408905 14594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:06.409165 14586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:06.409401 14586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:06.410591 14586 hybrid_clock.cc:648] HybridClock initialized: now 1790263266410580 us; error 28 us; skew 500 ppm
Sep 24 15:21:06 dist-test-slave-zwbb krb5kdc[13261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263266, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:06.412967 14586 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:21:06.413371 14586 webserver.cc:460] Webserver started at http://127.31.12.66:44903/ using document root <none> and password file <none>
I20260924 15:21:06.413601 14586 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:06.413668 14586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:06.414831 14586 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:21:06.415443 14603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:06.415618 14586 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:21:06.415716 14586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "23c27fce05f440d99056a2db20681a73"
format_stamp: "Formatted at 2026-09-24 15:21:04 on dist-test-slave-zwbb"
server_key: "18b601ae31e2afc2ff83b4390f2ed1d1"
server_key_iv: "66e56b913285f6720ef56593eadc4105"
server_key_version: "encryptionkey@0"
I20260924 15:21:06.416052 14586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:06.424151 14586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:06.426054 14586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:06.426204 14586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:06.426419 14586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:21:06.426714 14586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:21:06.426769 14586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:06.426808 14586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:21:06.426836 14586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:06.432616 14586 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:33433
I20260924 15:21:06.432660 14718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:33433 every 8 connection(s)
I20260924 15:21:06.432984 14586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:21:06.433990 14721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:06.433631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.66:50527)
0924 15:21:06.433779 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:06.433853 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
W20260924 15:21:06.434460 14719 heartbeater.cc:644] Failed to heartbeat to 127.31.12.126:33699 (0 consecutive failures): Network error: Failed to ping master at 127.31.12.126:33699: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
I20260924 15:21:06.434763 14721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:06.434694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.66:47661)
0924 15:21:06.434723 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:06.434747 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260924 15:21:06.435019 14721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:06.434970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.66:47609)
0924 15:21:06.434990 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:06.435008 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260924 15:21:06.435130 14719 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 15:21:06.443181 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 14586
I20260924 15:21:06.443424 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:41053
--local_ip_for_outbound_sockets=127.31.12.67
--tserver_master_addrs=127.31.12.126:33699
--webserver_port=39879
--webserver_interface=127.31.12.67
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:42991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:21:06.526953 14722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:06.527137 14722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:06.527159 14722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:06.528296 14722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:06.528337 14722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:06.528362 14722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:21:06.530051 14722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:42991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:41053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=39879
--tserver_master_addrs=127.31.12.126:33699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:06.530231 14722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:06.530400 14722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:06.535262 14729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:06.535255 14732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:06.535295 14722 server_base.cc:1034] running on GCE node
W20260924 15:21:06.535264 14730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:06.535653 14722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:06.535926 14722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:06.537127 14722 hybrid_clock.cc:648] HybridClock initialized: now 1790263266537113 us; error 25 us; skew 500 ppm
Sep 24 15:21:06 dist-test-slave-zwbb krb5kdc[13261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263266, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:06.539572 14722 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:21:06.539928 14722 webserver.cc:460] Webserver started at http://127.31.12.67:39879/ using document root <none> and password file <none>
I20260924 15:21:06.540115 14722 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:06.540155 14722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:06.541360 14722 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:21:06.541960 14739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:06.542133 14722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:06.542212 14722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "825a0a4a54e14c3dbcda29a6354c51b0"
format_stamp: "Formatted at 2026-09-24 15:21:04 on dist-test-slave-zwbb"
server_key: "a250e0d320f3a06e03e0309cb0ebe3ee"
server_key_iv: "6d5f6ab1f945218046111fcd2123bc9e"
server_key_version: "encryptionkey@0"
I20260924 15:21:06.542506 14722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:06.554917 14722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:06.557174 14722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:06.557286 14722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:06.557524 14722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:21:06.557802 14722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:21:06.557830 14722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:06.557852 14722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:21:06.557873 14722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:06.563570 14722 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:41053
I20260924 15:21:06.563639 14856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:41053 every 8 connection(s)
I20260924 15:21:06.563968 14722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:21:06.564996 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:06.564605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.67:38419)
0924 15:21:06.564760 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:06.564831 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
W20260924 15:21:06.565373 14857 heartbeater.cc:644] Failed to heartbeat to 127.31.12.126:33699 (0 consecutive failures): Network error: Failed to ping master at 127.31.12.126:33699: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
I20260924 15:21:06.565666 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:06.565569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.67:40837)
0924 15:21:06.565615 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:06.565640 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 15:21:06.565929 14859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:06.565847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.67:54865)
0924 15:21:06.565888 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:06.565910 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 15:21:06.566036 14857 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 15:21:06.568578 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 14722
I20260924 15:21:07.381744 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:07.381514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.65:44989)
0924 15:21:07.381647 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:07.381684 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":58,"threads_started":1}
I20260924 15:21:07.435657 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:07.435415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.66:54795)
0924 15:21:07.435559 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:07.435593 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260924 15:21:07.566504 14863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:07.566315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.67:47729)
0924 15:21:07.566429 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:07.566459 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
I20260924 15:21:07.927695 14864 raft_consensus.cc:482] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:21:07.927814 14864 raft_consensus.cc:504] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } } peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } }
I20260924 15:21:07.928073 14864 leader_election.cc:288] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 825a0a4a54e14c3dbcda29a6354c51b0 (127.31.12.67:41053), 23c27fce05f440d99056a2db20681a73 (127.31.12.66:33433)
Sep 24 15:21:07 dist-test-slave-zwbb krb5kdc[13261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263266, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.67@KRBTEST.COM
Sep 24 15:21:07 dist-test-slave-zwbb krb5kdc[13261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263266, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.66@KRBTEST.COM
I20260924 15:21:07.938722 14868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:07.928615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:44781 (local address 127.31.12.66:33433)
0924 15:21:07.928748 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:07.928756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:07.928783 (+ 27us) server_negotiation.cc:407] Connection header received
0924 15:21:07.929199 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:07.929201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:07.929275 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:07.929468 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:07.930124 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:07.931141 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:07.932132 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:07.932528 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:07.932637 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:07.935461 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:07.935468 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:07.935469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:07.935478 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:07.937096 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:07.937557 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:07.937560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:07.937560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:07.937645 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:07.937881 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:07.937882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:07.937882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:07.938105 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:07.938197 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:07.938504 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 15:21:07.938628 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260924 15:21:07.938732 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:07.928422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:33433 (local address 127.31.12.65:44781)
0924 15:21:07.928516 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:07.928524 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:07.928558 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:07.929422 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:07.929423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:07.929788 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:07.929980 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:07.929986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:07.931253 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:07.931254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:07.932034 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:07.932036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:07.932625 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:07.932626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:07.932747 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:07.933399 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:07.933410 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:07.935335 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:07.937230 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:07.937231 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:07.937231 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:07.937490 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:07.937720 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:07.937721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:07.937722 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:07.937805 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:07.938195 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:07.938203 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:07.938328 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 15:21:07.938434 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 15:21:07.938620 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":31,"thread_start_us":57,"threads_started":1}
I20260924 15:21:07.938760 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:07.928467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:52545 (local address 127.31.12.67:41053)
0924 15:21:07.928621 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:07.928636 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:07.928668 (+ 32us) server_negotiation.cc:407] Connection header received
0924 15:21:07.929104 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:07.929106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:07.929147 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:07.929264 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:07.930119 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:07.931103 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:07.932140 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:07.932540 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:07.932669 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:07.935159 (+ 2490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:07.935162 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:07.935163 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:07.935173 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:07.937001 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:07.937522 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:07.937525 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:07.937526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:07.937615 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:07.937885 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:07.937886 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:07.937886 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:07.938105 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:07.938197 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:07.938502 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 15:21:07.938647 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260924 15:21:07.938849 14865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:07.928247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:41053 (local address 127.31.12.65:52545)
0924 15:21:07.928357 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:07.928368 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:07.928426 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:07.929347 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:07.929350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:07.929768 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:07.929980 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:07.929986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:07.931196 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:07.931197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:07.932034 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:07.932036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:07.932654 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:07.932655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:07.932745 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:07.933517 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:07.933522 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:07.934966 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:07.937111 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:07.937119 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:07.937120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:07.937388 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:07.937692 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:07.937693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:07.937694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:07.937804 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:07.938192 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:07.938198 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:07.938328 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 15:21:07.938432 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 15:21:07.938703 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":165,"thread_start_us":64,"threads_started":1}
W20260924 15:21:07.939548 14468 leader_election.cc:341] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 825a0a4a54e14c3dbcda29a6354c51b0 (127.31.12.67:41053): Not found: Tablet not found: 48f4d274a359415382718dd00609e19b
W20260924 15:21:07.940045 14468 leader_election.cc:341] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 23c27fce05f440d99056a2db20681a73 (127.31.12.66:33433): Not found: Tablet not found: 48f4d274a359415382718dd00609e19b
I20260924 15:21:07.940107 14468 leader_election.cc:302] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9b26414cd9249ea8edceab73391e11e; no voters: 23c27fce05f440d99056a2db20681a73, 825a0a4a54e14c3dbcda29a6354c51b0
I20260924 15:21:07.940251 14864 raft_consensus.cc:2726] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:21:08.382308 14866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:08.382172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.65:38853)
0924 15:21:08.382233 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:08.382270 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 15:21:08.436347 14869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:08.436084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.66:59301)
0924 15:21:08.436230 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:08.436285 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":59,"threads_started":1}
I20260924 15:21:08.567144 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:08.566955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.67:46027)
0924 15:21:08.567077 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:08.567110 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260924 15:21:09.382965 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:09.382761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.65:51845)
0924 15:21:09.382885 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:09.382922 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260924 15:21:09.436976 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:09.436781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.66:54193)
0924 15:21:09.436897 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:09.436932 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260924 15:21:09.497753 14873 raft_consensus.cc:482] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:21:09.497843 14873 raft_consensus.cc:504] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } } peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } }
I20260924 15:21:09.497965 14873 leader_election.cc:288] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 825a0a4a54e14c3dbcda29a6354c51b0 (127.31.12.67:41053), 23c27fce05f440d99056a2db20681a73 (127.31.12.66:33433)
W20260924 15:21:09.498446 14468 leader_election.cc:341] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 23c27fce05f440d99056a2db20681a73 (127.31.12.66:33433): Not found: Tablet not found: 48f4d274a359415382718dd00609e19b
W20260924 15:21:09.498550 14468 leader_election.cc:341] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 825a0a4a54e14c3dbcda29a6354c51b0 (127.31.12.67:41053): Not found: Tablet not found: 48f4d274a359415382718dd00609e19b
I20260924 15:21:09.498588 14468 leader_election.cc:302] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9b26414cd9249ea8edceab73391e11e; no voters: 23c27fce05f440d99056a2db20681a73, 825a0a4a54e14c3dbcda29a6354c51b0
I20260924 15:21:09.498672 14873 raft_consensus.cc:2726] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:21:09.567768 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:09.567559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.67:43007)
0924 15:21:09.567692 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:09.567731 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260924 15:21:10.383606 14875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:10.383388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.65:55047)
0924 15:21:10.383525 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:10.383562 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":71,"threads_started":1}
I20260924 15:21:10.437711 14876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:10.437507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.66:41119)
0924 15:21:10.437645 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:10.437676 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260924 15:21:10.568375 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:10.568187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.67:59355)
0924 15:21:10.568299 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:10.568334 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":55,"threads_started":1}
I20260924 15:21:11.384326 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:11.384116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.65:53481)
0924 15:21:11.384243 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:11.384282 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260924 15:21:11.438442 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:11.438165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.66:46685)
0924 15:21:11.438361 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:11.438402 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":108,"threads_started":1}
I20260924 15:21:11.569126 14881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:11.568878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.67:42053)
0924 15:21:11.569044 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:11.569083 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":76,"threads_started":1}
I20260924 15:21:12.137681 14882 raft_consensus.cc:482] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:21:12.137789 14882 raft_consensus.cc:504] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } } peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } }
I20260924 15:21:12.137998 14882 leader_election.cc:288] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 825a0a4a54e14c3dbcda29a6354c51b0 (127.31.12.67:41053), 23c27fce05f440d99056a2db20681a73 (127.31.12.66:33433)
W20260924 15:21:12.138415 14468 leader_election.cc:341] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 23c27fce05f440d99056a2db20681a73 (127.31.12.66:33433): Not found: Tablet not found: 48f4d274a359415382718dd00609e19b
W20260924 15:21:12.138494 14468 leader_election.cc:341] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 825a0a4a54e14c3dbcda29a6354c51b0 (127.31.12.67:41053): Not found: Tablet not found: 48f4d274a359415382718dd00609e19b
I20260924 15:21:12.138510 14468 leader_election.cc:302] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9b26414cd9249ea8edceab73391e11e; no voters: 23c27fce05f440d99056a2db20681a73, 825a0a4a54e14c3dbcda29a6354c51b0
I20260924 15:21:12.138588 14882 raft_consensus.cc:2726] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:21:12.385072 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:12.384837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.65:49097)
0924 15:21:12.384982 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:12.385024 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":60,"threads_started":1}
I20260924 15:21:12.439170 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:12.438936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.66:32869)
0924 15:21:12.439086 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:12.439128 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":63,"threads_started":1}
I20260924 15:21:12.569841 14885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:12.569651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.67:57719)
0924 15:21:12.569774 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:12.569801 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":65,"threads_started":1}
I20260924 15:21:13.385763 14886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:13.385559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.65:52533)
0924 15:21:13.385683 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:13.385719 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
I20260924 15:21:13.439783 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:13.439591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.66:46777)
0924 15:21:13.439715 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:13.439746 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
I20260924 15:21:13.570508 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:13.570303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.67:47395)
0924 15:21:13.570435 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:13.570471 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260924 15:21:14.386407 14889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:14.386183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.65:42947)
0924 15:21:14.386321 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:14.386360 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260924 15:21:14.440428 14890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:14.440236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.66:58695)
0924 15:21:14.440358 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:14.440391 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260924 15:21:14.571170 14891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:14.570934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.67:56765)
0924 15:21:14.571085 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:14.571120 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
I20260924 15:21:15.387061 14892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:15.386839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.65:35969)
0924 15:21:15.386977 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:15.387014 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260924 15:21:15.441017 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:15.440833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.66:39163)
0924 15:21:15.440953 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:15.440982 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260924 15:21:15.571889 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:15.571654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.67:33839)
0924 15:21:15.571796 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:15.571840 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260924 15:21:16.385035 14896 raft_consensus.cc:482] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:21:16.385128 14896 raft_consensus.cc:504] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } } peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } }
I20260924 15:21:16.385289 14896 leader_election.cc:288] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 825a0a4a54e14c3dbcda29a6354c51b0 (127.31.12.67:41053), 23c27fce05f440d99056a2db20681a73 (127.31.12.66:33433)
W20260924 15:21:16.385716 14468 leader_election.cc:341] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 23c27fce05f440d99056a2db20681a73 (127.31.12.66:33433): Not found: Tablet not found: 48f4d274a359415382718dd00609e19b
W20260924 15:21:16.385792 14468 leader_election.cc:341] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 825a0a4a54e14c3dbcda29a6354c51b0 (127.31.12.67:41053): Not found: Tablet not found: 48f4d274a359415382718dd00609e19b
I20260924 15:21:16.385808 14468 leader_election.cc:302] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9b26414cd9249ea8edceab73391e11e; no voters: 23c27fce05f440d99056a2db20681a73, 825a0a4a54e14c3dbcda29a6354c51b0
I20260924 15:21:16.385905 14896 raft_consensus.cc:2726] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:21:16.387681 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:16.387501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.65:54233)
0924 15:21:16.387612 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:16.387646 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260924 15:21:16.441771 14898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:16.441568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.66:44047)
0924 15:21:16.441702 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:16.441736 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":67,"threads_started":1}
I20260924 15:21:16.572582 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:16.572333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.67:34771)
0924 15:21:16.572475 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:16.572511 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260924 15:21:17.388345 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:17.388142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.65:41245)
0924 15:21:17.388263 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:17.388299 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260924 15:21:17.442466 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:17.442204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.66:44053)
0924 15:21:17.442357 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:17.442417 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":62,"threads_started":1}
I20260924 15:21:17.573338 14902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:17.573090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.67:38483)
0924 15:21:17.573247 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:17.573289 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260924 15:21:18.389056 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:18.388823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.65:57885)
0924 15:21:18.388965 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:18.389009 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260924 15:21:18.443125 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:18.442933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.66:54055)
0924 15:21:18.443047 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:18.443084 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":58,"threads_started":1}
I20260924 15:21:18.532618 14905 raft_consensus.cc:482] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:21:18.532718 14905 raft_consensus.cc:504] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } } peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } }
I20260924 15:21:18.532848 14905 leader_election.cc:288] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 825a0a4a54e14c3dbcda29a6354c51b0 (127.31.12.67:41053), 23c27fce05f440d99056a2db20681a73 (127.31.12.66:33433)
W20260924 15:21:18.533352 14468 leader_election.cc:341] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 23c27fce05f440d99056a2db20681a73 (127.31.12.66:33433): Not found: Tablet not found: 48f4d274a359415382718dd00609e19b
W20260924 15:21:18.533420 14468 leader_election.cc:341] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 825a0a4a54e14c3dbcda29a6354c51b0 (127.31.12.67:41053): Not found: Tablet not found: 48f4d274a359415382718dd00609e19b
I20260924 15:21:18.533499 14468 leader_election.cc:302] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9b26414cd9249ea8edceab73391e11e; no voters: 23c27fce05f440d99056a2db20681a73, 825a0a4a54e14c3dbcda29a6354c51b0
I20260924 15:21:18.533599 14905 raft_consensus.cc:2726] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:21:18.574105 14906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:18.573890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.67:52671)
0924 15:21:18.574025 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:18.574061 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260924 15:21:19.389690 14907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:19.389486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.65:48273)
0924 15:21:19.389607 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:19.389644 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
I20260924 15:21:19.443754 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:19.443565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.66:56603)
0924 15:21:19.443685 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:19.443716 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260924 15:21:19.574791 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:19.574567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.67:42451)
0924 15:21:19.574704 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:19.574744 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260924 15:21:20.390267 14910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:20.390075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.65:37297)
0924 15:21:20.390188 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:20.390223 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":57,"threads_started":1}
I20260924 15:21:20.444331 14911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:20.444139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.66:50407)
0924 15:21:20.444255 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:20.444289 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":60,"threads_started":1}
I20260924 15:21:20.575475 14912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:20.575251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.67:43853)
0924 15:21:20.575389 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:20.575428 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260924 15:21:20.857746 14913 raft_consensus.cc:482] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:21:20.857861 14913 raft_consensus.cc:504] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "825a0a4a54e14c3dbcda29a6354c51b0" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41053 } } peers { permanent_uuid: "23c27fce05f440d99056a2db20681a73" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33433 } } peers { permanent_uuid: "f9b26414cd9249ea8edceab73391e11e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 43305 } }
I20260924 15:21:20.858042 14913 leader_election.cc:288] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 825a0a4a54e14c3dbcda29a6354c51b0 (127.31.12.67:41053), 23c27fce05f440d99056a2db20681a73 (127.31.12.66:33433)
W20260924 15:21:20.858415 14468 leader_election.cc:341] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 825a0a4a54e14c3dbcda29a6354c51b0 (127.31.12.67:41053): Not found: Tablet not found: 48f4d274a359415382718dd00609e19b
W20260924 15:21:20.858491 14468 leader_election.cc:341] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 23c27fce05f440d99056a2db20681a73 (127.31.12.66:33433): Not found: Tablet not found: 48f4d274a359415382718dd00609e19b
I20260924 15:21:20.858507 14468 leader_election.cc:302] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9b26414cd9249ea8edceab73391e11e; no voters: 23c27fce05f440d99056a2db20681a73, 825a0a4a54e14c3dbcda29a6354c51b0
I20260924 15:21:20.858580 14913 raft_consensus.cc:2726] T 48f4d274a359415382718dd00609e19b P f9b26414cd9249ea8edceab73391e11e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:21:21.390987 14914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:21.390756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.65:39649)
0924 15:21:21.390902 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:21.390942 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
I20260924 15:21:21.445078 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:21.444777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.66:47209)
0924 15:21:21.445005 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:21.445039 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":82,"threads_started":1}
I20260924 15:21:21.569283 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 14419
I20260924 15:21:21.574565 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 14586
I20260924 15:21:21.576192 14916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:21.575967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33699 (local address 127.31.12.67:33001)
0924 15:21:21.576093 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:21.576154 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:33699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260924 15:21:21.579320 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 14722
2026-09-24T15:21:21Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18596 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:21:21 dist-test-slave-zwbb krb5kdc[14919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:21:21 dist-test-slave-zwbb krb5kdc[14919](info): set up 2 sockets
Sep 24 15:21:21 dist-test-slave-zwbb krb5kdc[14919](info): commencing operation
krb5kdc: starting...
W20260924 15:21:22.619287 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:21:22 dist-test-slave-zwbb krb5kdc[14919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:21:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:21:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:21:22.739960 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:38891
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:38891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:21:22.820194 14932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:22.820415 14932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:22.820452 14932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:22.821579 14932 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 15:21:22.821643 14932 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 15:21:22.821666 14932 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 15:21:22.821698 14932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:21:22.821729 14932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:22.821754 14932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:22.821784 14932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:21:22.821812 14932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:21:22.823268 14932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:38891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:38891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14932
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:22.823529 14932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:22.823738 14932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:22.828863 14940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:22.828851 14937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:22.828912 14938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:22.829003 14932 server_base.cc:1034] running on GCE node
I20260924 15:21:22.829295 14932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:22.829602 14932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:22.830760 14932 hybrid_clock.cc:648] HybridClock initialized: now 1790263282830749 us; error 33 us; skew 500 ppm
Sep 24 15:21:22 dist-test-slave-zwbb krb5kdc[14919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263282, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:22.833153 14932 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:21:22.833652 14932 webserver.cc:460] Webserver started at http://127.31.12.126:34303/ using document root <none> and password file <none>
I20260924 15:21:22.833885 14932 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:22.833957 14932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:22.834100 14932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:21:22.835083 14932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "6b543a3dbe1a4175ba59673d0823c960"
format_stamp: "Formatted at 2026-09-24 15:21:22 on dist-test-slave-zwbb"
server_key: "807003b2459e6e8fe525a8d962a4d694"
server_key_iv: "62d2f80a3ff272e713562caa880432a9"
server_key_version: "encryptionkey@0"
I20260924 15:21:22.835381 14932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "6b543a3dbe1a4175ba59673d0823c960"
format_stamp: "Formatted at 2026-09-24 15:21:22 on dist-test-slave-zwbb"
server_key: "807003b2459e6e8fe525a8d962a4d694"
server_key_iv: "62d2f80a3ff272e713562caa880432a9"
server_key_version: "encryptionkey@0"
I20260924 15:21:22.836614 14932 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:21:22.837319 14948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:22.837527 14932 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:21:22.837622 14932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "6b543a3dbe1a4175ba59673d0823c960"
format_stamp: "Formatted at 2026-09-24 15:21:22 on dist-test-slave-zwbb"
server_key: "807003b2459e6e8fe525a8d962a4d694"
server_key_iv: "62d2f80a3ff272e713562caa880432a9"
server_key_version: "encryptionkey@0"
I20260924 15:21:22.837710 14932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:22.876211 14932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:22.878342 14932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:22.878512 14932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:22.882488 14932 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:38891
I20260924 15:21:22.882524 15001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:38891 every 8 connection(s)
I20260924 15:21:22.882908 14932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:21:22.883785 15002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:21:22.884732 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 14932
I20260924 15:21:22.884831 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:21:22.885046 31793 external_mini_cluster.cc:1428] Setting key aa5a29986fb444a5cf0f82f3488efcbe
I20260924 15:21:22.886411 15002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960: Bootstrap starting.
I20260924 15:21:22.887205 15002 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960: Neither blocks nor log segments found. Creating new log.
I20260924 15:21:22.887531 15002 log.cc:824] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:22.888257 15002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960: No bootstrap required, opened a new log
I20260924 15:21:22.890693 15002 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b543a3dbe1a4175ba59673d0823c960" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38891 } }
I20260924 15:21:22.890852 15002 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:21:22.890905 15002 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b543a3dbe1a4175ba59673d0823c960, State: Initialized, Role: FOLLOWER
I20260924 15:21:22.891027 15002 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b543a3dbe1a4175ba59673d0823c960" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38891 } }
I20260924 15:21:22.891103 15002 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:21:22.891124 15002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:21:22.891192 15002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:21:22.891800 15002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b543a3dbe1a4175ba59673d0823c960" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38891 } }
I20260924 15:21:22.891907 15002 leader_election.cc:302] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b543a3dbe1a4175ba59673d0823c960; no voters:
I20260924 15:21:22.892079 15002 leader_election.cc:288] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:21:22.892143 15007 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 1 FOLLOWER]: Leader election won for term 1
Sep 24 15:21:22 dist-test-slave-zwbb krb5kdc[14919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:21:22.892320 15007 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 1 LEADER]: Becoming Leader. State: Replica: 6b543a3dbe1a4175ba59673d0823c960, State: Running, Role: LEADER
I20260924 15:21:22.892357 15002 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:21:22.892397 15007 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b543a3dbe1a4175ba59673d0823c960" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38891 } }
I20260924 15:21:22.892726 15007 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b543a3dbe1a4175ba59673d0823c960. Latest consensus state: current_term: 1 leader_uuid: "6b543a3dbe1a4175ba59673d0823c960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b543a3dbe1a4175ba59673d0823c960" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38891 } } }
I20260924 15:21:22.892765 15008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b543a3dbe1a4175ba59673d0823c960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b543a3dbe1a4175ba59673d0823c960" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38891 } } }
I20260924 15:21:22.892797 15007 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [sys.catalog]: This master's current role is: LEADER
I20260924 15:21:22.893057 15014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:21:22.892809 15008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [sys.catalog]: This master's current role is: LEADER
I20260924 15:21:22.893601 15014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:21:22.896549 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:22.885592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56826 (local address 127.31.12.126:38891)
0924 15:21:22.885768 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:22.885775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:22.885799 (+ 24us) server_negotiation.cc:407] Connection header received
0924 15:21:22.886224 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:22.886230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:22.886273 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:22.886384 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:22.887196 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:22.888299 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:22.889548 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:22.889959 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:22.890098 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:22.892869 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:22.892873 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:22.892873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:22.892885 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:22.894817 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:22.895353 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:22.895375 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:22.895375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:22.895479 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:22.895667 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:22.895667 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:22.895668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:22.895959 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:22.896131 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:22.896197 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 15:21:22.896331 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260924 15:21:22.898497 15014 catalog_manager.cc:1347] Generated new cluster ID: e475393132694bdca06419820774abaa
I20260924 15:21:22.898557 15014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:21:22.907799 15014 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:21:22.908293 15014 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:21:22.912151 15014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960: Generated new TSK 0
I20260924 15:21:22.912321 15014 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
I20260924 15:21:22.941329 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:38891
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:21:23.019675 15029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:23.019865 15029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:23.019888 15029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:23.021034 15029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:23.021073 15029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:23.021099 15029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:21:23.022549 15029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:38891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:23.022722 15029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:23.022948 15029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:23.027798 15034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:23.027837 15037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:23.027807 15035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:23.027840 15029 server_base.cc:1034] running on GCE node
I20260924 15:21:23.028162 15029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:23.028404 15029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:23.029569 15029 hybrid_clock.cc:648] HybridClock initialized: now 1790263283029563 us; error 36 us; skew 500 ppm
Sep 24 15:21:23 dist-test-slave-zwbb krb5kdc[14919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263283, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:23.031908 15029 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:21:23.032310 15029 webserver.cc:460] Webserver started at http://127.31.12.65:34099/ using document root <none> and password file <none>
I20260924 15:21:23.032536 15029 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:23.032604 15029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:23.032755 15029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:21:23.033840 15029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "c8795445a8444291aaf5234a6f1ca235"
format_stamp: "Formatted at 2026-09-24 15:21:23 on dist-test-slave-zwbb"
server_key: "dfd4ee048e2f19e023497f537bd3292a"
server_key_iv: "875d18effd49d0509d57fab07ce9bc37"
server_key_version: "encryptionkey@0"
I20260924 15:21:23.034212 15029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "c8795445a8444291aaf5234a6f1ca235"
format_stamp: "Formatted at 2026-09-24 15:21:23 on dist-test-slave-zwbb"
server_key: "dfd4ee048e2f19e023497f537bd3292a"
server_key_iv: "875d18effd49d0509d57fab07ce9bc37"
server_key_version: "encryptionkey@0"
I20260924 15:21:23.035553 15029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:21:23.036254 15044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:23.036429 15029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:21:23.036535 15029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "c8795445a8444291aaf5234a6f1ca235"
format_stamp: "Formatted at 2026-09-24 15:21:23 on dist-test-slave-zwbb"
server_key: "dfd4ee048e2f19e023497f537bd3292a"
server_key_iv: "875d18effd49d0509d57fab07ce9bc37"
server_key_version: "encryptionkey@0"
I20260924 15:21:23.036617 15029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:23.054500 15029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:23.056438 15029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:23.056604 15029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:23.056847 15029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:21:23.057183 15029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:21:23.057238 15029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:23.057286 15029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:21:23.057309 15029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:23.063102 15029 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:42095
I20260924 15:21:23.063148 15159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:42095 every 8 connection(s)
I20260924 15:21:23.063457 15029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:21:23.065991 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 15029
I20260924 15:21:23.066112 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:21:23.066344 31793 external_mini_cluster.cc:1428] Setting key f5fec42ea40533ca0963557951f90300
Sep 24 15:21:23 dist-test-slave-zwbb krb5kdc[14919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263283, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:21:23.074043 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:23.064151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:35891 (local address 127.31.12.126:38891)
0924 15:21:23.064222 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:23.064224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:23.064421 (+ 197us) server_negotiation.cc:407] Connection header received
0924 15:21:23.064948 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:23.064949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:23.064958 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:23.065018 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:23.066264 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.066976 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.067995 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.068330 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.068402 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:23.070894 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:23.070896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:23.070896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:23.070905 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:23.072603 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:23.072988 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:23.072990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:23.072990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:23.073056 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:23.073306 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:23.073307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:23.073307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:23.073472 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:23.073562 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:23.073833 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 15:21:23.073931 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 15:21:23.074051 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:23.064120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.65:35891)
0924 15:21:23.064305 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:23.064346 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:23.064831 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:23.065108 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:23.065111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:23.065777 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:23.065988 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:23.065994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.067078 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.067079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:23.067899 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:23.067901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.068441 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.068442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:23.068616 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:23.069146 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:23.069157 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:23.070794 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:23.072691 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:23.072698 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:23.072700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:23.072912 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:23.073128 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:23.073128 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:23.073129 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:23.073240 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:23.073563 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:23.073569 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:23.073691 (+ 122us) client_negotiation.cc:769] Sending connection context
0924 15:21:23.073777 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 15:21:23.073896 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":112,"threads_started":1}
I20260924 15:21:23.074622 15160 heartbeater.cc:347] Connected to a master server at 127.31.12.126:38891
I20260924 15:21:23.074712 15160 heartbeater.cc:464] Registering TS with master...
I20260924 15:21:23.074905 15160 heartbeater.cc:511] Master 127.31.12.126:38891 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.12.66@KRBTEST.COM; defaulting to no policy
I20260924 15:21:23.075485 14965 ts_manager.cc:194] Registered new tserver with Master: c8795445a8444291aaf5234a6f1ca235 (127.31.12.65:42095)
I20260924 15:21:23.076099 14965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:35891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
I20260924 15:21:23.092388 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:38891
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:21:23.171528 15165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:23.171765 15165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:23.171803 15165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:23.172930 15165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:23.173007 15165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:23.173054 15165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:21:23.174486 15165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:38891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:23.174751 15165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:23.174989 15165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:23.180034 15170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:23.180059 15173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:23.180060 15171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:23.180330 15165 server_base.cc:1034] running on GCE node
I20260924 15:21:23.180521 15165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:23.180712 15165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:23.181846 15165 hybrid_clock.cc:648] HybridClock initialized: now 1790263283181821 us; error 39 us; skew 500 ppm
Sep 24 15:21:23 dist-test-slave-zwbb krb5kdc[14919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263283, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:23.184136 15165 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:21:23.184494 15165 webserver.cc:460] Webserver started at http://127.31.12.66:38315/ using document root <none> and password file <none>
I20260924 15:21:23.184669 15165 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:23.184710 15165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:23.184794 15165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:21:23.185838 15165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "0dde3faf177d4fcbb16905769398d7fd"
format_stamp: "Formatted at 2026-09-24 15:21:23 on dist-test-slave-zwbb"
server_key: "e9593341322ff8e1f86dd06834ebd980"
server_key_iv: "8dd3239116d4fbc18db82283e5b568c3"
server_key_version: "encryptionkey@0"
I20260924 15:21:23.186146 15165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "0dde3faf177d4fcbb16905769398d7fd"
format_stamp: "Formatted at 2026-09-24 15:21:23 on dist-test-slave-zwbb"
server_key: "e9593341322ff8e1f86dd06834ebd980"
server_key_iv: "8dd3239116d4fbc18db82283e5b568c3"
server_key_version: "encryptionkey@0"
I20260924 15:21:23.187498 15165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:21:23.188177 15180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:23.188344 15165 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:21:23.188443 15165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "0dde3faf177d4fcbb16905769398d7fd"
format_stamp: "Formatted at 2026-09-24 15:21:23 on dist-test-slave-zwbb"
server_key: "e9593341322ff8e1f86dd06834ebd980"
server_key_iv: "8dd3239116d4fbc18db82283e5b568c3"
server_key_version: "encryptionkey@0"
I20260924 15:21:23.188526 15165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:23.196607 15165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:23.198421 15165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:23.198576 15165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:23.198807 15165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:21:23.199097 15165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:21:23.199151 15165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:23.199189 15165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:21:23.199215 15165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:23.204710 15165 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:33753
I20260924 15:21:23.204748 15298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:33753 every 8 connection(s)
I20260924 15:21:23.205097 15165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:21:23.207144 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 15165
I20260924 15:21:23.207264 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:21:23.207471 31793 external_mini_cluster.cc:1428] Setting key c373196b1805d2cbd247fa421ec1f3aa
Sep 24 15:21:23 dist-test-slave-zwbb krb5kdc[14919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263283, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:21:23.214704 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:23.205828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:33045 (local address 127.31.12.126:38891)
0924 15:21:23.205876 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:23.205878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:23.206053 (+ 175us) server_negotiation.cc:407] Connection header received
0924 15:21:23.206504 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:23.206505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:23.206515 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:23.206568 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:23.207564 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.208284 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.209255 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.209626 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.209701 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:23.211889 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:23.211890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:23.211891 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:23.211898 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:23.213288 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:23.213742 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:23.213744 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:23.213744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:23.213810 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:23.214061 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:23.214062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:23.214063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:23.214182 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:23.214265 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:23.214526 (+ 261us) server_negotiation.cc:299] Negotiation successful
0924 15:21:23.214611 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:21:23.214890 15302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:23.205775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.66:33045)
0924 15:21:23.205928 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:23.205967 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:23.206421 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:23.206646 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:23.206649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:23.207184 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:23.207434 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:23.207440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.208371 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.208372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:23.209147 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:23.209148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.209720 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.209721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:23.209869 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:23.210334 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:23.210345 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:23.211793 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:23.213387 (+ 1594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:23.213395 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:23.213397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:23.213655 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:23.213876 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:23.213877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:23.213877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:23.213990 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:23.214255 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:23.214262 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:23.214397 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 15:21:23.214505 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 15:21:23.214667 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260924 15:21:23.215333 15299 heartbeater.cc:347] Connected to a master server at 127.31.12.126:38891
I20260924 15:21:23.215436 15299 heartbeater.cc:464] Registering TS with master...
I20260924 15:21:23.215636 15299 heartbeater.cc:511] Master 127.31.12.126:38891 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.31.12.67@KRBTEST.COM; defaulting to no policy
I20260924 15:21:23.216006 14965 ts_manager.cc:194] Registered new tserver with Master: 0dde3faf177d4fcbb16905769398d7fd (127.31.12.66:33753)
I20260924 15:21:23.216368 14965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:33045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
I20260924 15:21:23.232443 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:38891
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:21:23.312883 15305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:23.313083 15305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:23.313107 15305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:23.314181 15305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:23.314216 15305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:23.314242 15305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:21:23.315593 15305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:38891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:23.315773 15305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:23.315935 15305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:23.320799 15310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:23.320776 15313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:23.320783 15311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:23.321053 15305 server_base.cc:1034] running on GCE node
I20260924 15:21:23.321260 15305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:23.321532 15305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:23.322697 15305 hybrid_clock.cc:648] HybridClock initialized: now 1790263283322687 us; error 41 us; skew 500 ppm
Sep 24 15:21:23 dist-test-slave-zwbb krb5kdc[14919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263283, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:23.325083 15305 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:21:23.325522 15305 webserver.cc:460] Webserver started at http://127.31.12.67:46271/ using document root <none> and password file <none>
I20260924 15:21:23.325730 15305 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:23.325770 15305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:23.325858 15305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:21:23.326828 15305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "0c339766ca354ae88b148dc07f93b0da"
format_stamp: "Formatted at 2026-09-24 15:21:23 on dist-test-slave-zwbb"
server_key: "1eabcafc97b76446cf215ad1cdd19a3f"
server_key_iv: "396a3591f02a627a588eb0b1a2729b76"
server_key_version: "encryptionkey@0"
I20260924 15:21:23.327131 15305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "0c339766ca354ae88b148dc07f93b0da"
format_stamp: "Formatted at 2026-09-24 15:21:23 on dist-test-slave-zwbb"
server_key: "1eabcafc97b76446cf215ad1cdd19a3f"
server_key_iv: "396a3591f02a627a588eb0b1a2729b76"
server_key_version: "encryptionkey@0"
I20260924 15:21:23.328377 15305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:21:23.329047 15320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:23.329244 15305 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:21:23.329349 15305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "0c339766ca354ae88b148dc07f93b0da"
format_stamp: "Formatted at 2026-09-24 15:21:23 on dist-test-slave-zwbb"
server_key: "1eabcafc97b76446cf215ad1cdd19a3f"
server_key_iv: "396a3591f02a627a588eb0b1a2729b76"
server_key_version: "encryptionkey@0"
I20260924 15:21:23.329465 15305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:23.360986 15305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:23.363109 15305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:23.363300 15305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:23.363557 15305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:21:23.363885 15305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:21:23.363941 15305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:23.363988 15305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:21:23.364012 15305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:23.370959 15305 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:33089
I20260924 15:21:23.371052 15436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:33089 every 8 connection(s)
I20260924 15:21:23.371413 15305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:21:23 dist-test-slave-zwbb krb5kdc[14919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263283, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:21:23.377594 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 15305
I20260924 15:21:23.377705 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:21:23.377902 31793 external_mini_cluster.cc:1428] Setting key 3481e0d6bd9d4e6ce50b70fbe7fbb015
I20260924 15:21:23.380575 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:23.372236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:55667 (local address 127.31.12.126:38891)
0924 15:21:23.372303 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:23.372305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:23.372503 (+ 198us) server_negotiation.cc:407] Connection header received
0924 15:21:23.373000 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:23.373001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:23.373011 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:23.373066 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:23.374061 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.374558 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.375477 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.375660 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.375709 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:23.377947 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:23.377949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:23.377949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:23.377957 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:23.379414 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:23.379804 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:23.379805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:23.379805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:23.379855 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:23.380041 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:23.380042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:23.380042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:23.380148 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:23.380199 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:23.380446 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 15:21:23.380504 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 15:21:23.380728 15439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:23.372197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.67:55667)
0924 15:21:23.372373 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:23.372416 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:23.372908 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:23.373129 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:23.373133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:23.373784 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:23.373983 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:23.373988 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.374639 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.374639 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:23.375404 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:23.375406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.375731 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.375732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:23.375828 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:23.376311 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:23.376321 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:23.377840 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:23.379522 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:23.379527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:23.379529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:23.379742 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:23.379902 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:23.379903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:23.379903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:23.379988 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:23.380204 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:23.380208 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:23.380316 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 15:21:23.380418 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 15:21:23.380541 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
I20260924 15:21:23.381198 15437 heartbeater.cc:347] Connected to a master server at 127.31.12.126:38891
I20260924 15:21:23.381275 15437 heartbeater.cc:464] Registering TS with master...
I20260924 15:21:23.381510 15437 heartbeater.cc:511] Master 127.31.12.126:38891 requested a full tablet report, sending...
I20260924 15:21:23.381884 14965 ts_manager.cc:194] Registered new tserver with Master: 0c339766ca354ae88b148dc07f93b0da (127.31.12.67:33089)
I20260924 15:21:23.382280 14965 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:55667
I20260924 15:21:23.389580 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:21:23.396987 15005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:23.390869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56828 (local address 127.31.12.126:38891)
0924 15:21:23.390938 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:23.390940 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:23.391040 (+ 100us) server_negotiation.cc:407] Connection header received
0924 15:21:23.391103 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:23.391104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:23.391115 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:23.391189 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:23.391714 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.392207 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.393104 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.393318 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.393373 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:23.394619 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:23.394620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:23.394620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:23.394626 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:23.395732 (+ 1106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:23.396103 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:23.396104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:23.396105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:23.396168 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:23.396388 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:23.396389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:23.396389 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:23.396553 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:23.396663 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:23.396795 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 15:21:23.396880 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 15:21:23.398415 14965 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:21:23.399983 14965 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:21:23.406639 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:23.402184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:42095 (local address 127.0.0.1:43784)
0924 15:21:23.402509 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:23.402548 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:23.402613 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:23.402984 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:23.402984 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:23.402987 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:23.403033 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:23.403034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.404285 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.404286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:23.405614 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:23.405617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.406378 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.406379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:23.406442 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:23.406448 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 15:21:23.406498 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 15:21:23.406538 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":47,"thread_start_us":69,"threads_started":1}
I20260924 15:21:23.406790 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:23.402317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58442 (local address 127.31.12.66:33753)
0924 15:21:23.402503 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:23.402508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:23.402626 (+ 118us) server_negotiation.cc:407] Connection header received
0924 15:21:23.402702 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:23.402704 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:23.402721 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:23.402795 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:23.403044 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.404164 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.405698 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.406432 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.406505 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:23.406569 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:23.406653 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 15:21:23.406699 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":72,"threads_started":1}
I20260924 15:21:23.406855 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:23.402518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49942 (local address 127.31.12.67:33089)
0924 15:21:23.403201 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:23.403205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:23.403220 (+ 15us) server_negotiation.cc:407] Connection header received
0924 15:21:23.403249 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:23.403250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:23.403262 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:23.403339 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:23.403464 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.404133 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.405354 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.406116 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.406193 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:23.406251 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:23.406590 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 15:21:23.406644 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":99,"threads_started":1}
I20260924 15:21:23.406948 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:23.402286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43784 (local address 127.31.12.65:42095)
0924 15:21:23.402749 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:23.402761 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:23.402770 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:21:23.402828 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:23.402830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:23.402845 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:23.402925 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:23.403124 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.404184 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.405715 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.406289 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.406367 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:23.406424 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:23.406818 (+ 394us) server_negotiation.cc:299] Negotiation successful
0924 15:21:23.406867 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":103,"threads_started":1}
I20260924 15:21:23.406742 15445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:23.402188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:33753 (local address 127.0.0.1:58442)
0924 15:21:23.402508 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:23.402554 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:23.402613 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:23.402836 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:23.402840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:23.402841 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:23.402912 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:23.402913 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.404284 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.404285 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:23.405615 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:23.405617 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.406507 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.406507 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:23.406570 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:23.406574 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:21:23.406638 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 15:21:23.406674 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":74,"spinlock_wait_cycles":1280,"thread_start_us":89,"threads_started":1}
I20260924 15:21:23.406526 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:23.402404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:33089 (local address 127.0.0.1:49942)
0924 15:21:23.403037 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:23.403046 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:23.403088 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:23.403349 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:23.403350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:23.403352 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:23.403412 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:23.403413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.404233 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.404234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:23.405269 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:23.405271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.406213 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.406214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:23.406304 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:23.406321 (+ 17us) client_negotiation.cc:769] Sending connection context
0924 15:21:23.406387 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 15:21:23.406446 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":39,"threads_started":1}
I20260924 15:21:23.407684 15093 tablet_service.cc:1467] Processing CreateTablet for tablet ee22ddfe7d784fb38e89a1cd073b0494 (DEFAULT_TABLE table=test-table [id=e4a8a6de18d940fd89e85b5033958a54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:21:23.407795 15232 tablet_service.cc:1467] Processing CreateTablet for tablet ee22ddfe7d784fb38e89a1cd073b0494 (DEFAULT_TABLE table=test-table [id=e4a8a6de18d940fd89e85b5033958a54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:21:23.408066 15093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee22ddfe7d784fb38e89a1cd073b0494. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:21:23.408164 15232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee22ddfe7d784fb38e89a1cd073b0494. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:21:23.410110 15368 tablet_service.cc:1467] Processing CreateTablet for tablet ee22ddfe7d784fb38e89a1cd073b0494 (DEFAULT_TABLE table=test-table [id=e4a8a6de18d940fd89e85b5033958a54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:21:23.410424 15368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee22ddfe7d784fb38e89a1cd073b0494. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:21:23.410841 15451 tablet_bootstrap.cc:492] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd: Bootstrap starting.
I20260924 15:21:23.411257 15452 tablet_bootstrap.cc:492] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235: Bootstrap starting.
I20260924 15:21:23.411465 15451 tablet_bootstrap.cc:654] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd: Neither blocks nor log segments found. Creating new log.
I20260924 15:21:23.411960 15451 log.cc:824] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:23.411971 15452 tablet_bootstrap.cc:654] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235: Neither blocks nor log segments found. Creating new log.
I20260924 15:21:23.412544 15452 log.cc:824] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:23.412966 15451 tablet_bootstrap.cc:492] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd: No bootstrap required, opened a new log
I20260924 15:21:23.413074 15451 ts_tablet_manager.cc:1397] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:21:23.413348 15452 tablet_bootstrap.cc:492] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235: No bootstrap required, opened a new log
I20260924 15:21:23.413421 15452 ts_tablet_manager.cc:1397] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:21:23.413730 15455 tablet_bootstrap.cc:492] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da: Bootstrap starting.
I20260924 15:21:23.414394 15455 tablet_bootstrap.cc:654] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da: Neither blocks nor log segments found. Creating new log.
I20260924 15:21:23.414911 15452 raft_consensus.cc:348] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } }
I20260924 15:21:23.415022 15455 log.cc:824] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:23.415076 15452 raft_consensus.cc:374] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:21:23.415140 15452 raft_consensus.cc:729] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8795445a8444291aaf5234a6f1ca235, State: Initialized, Role: FOLLOWER
I20260924 15:21:23.415266 15452 consensus_queue.cc:260] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } }
I20260924 15:21:23.415263 15451 raft_consensus.cc:348] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } }
I20260924 15:21:23.415441 15451 raft_consensus.cc:374] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:21:23.415488 15451 raft_consensus.cc:729] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dde3faf177d4fcbb16905769398d7fd, State: Initialized, Role: FOLLOWER
I20260924 15:21:23.415611 15160 heartbeater.cc:503] Master 127.31.12.126:38891 was elected leader, sending a full tablet report...
I20260924 15:21:23.415653 15451 consensus_queue.cc:260] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } }
I20260924 15:21:23.415602 15452 ts_tablet_manager.cc:1428] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:21:23.415931 15451 ts_tablet_manager.cc:1428] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:21:23.416045 15299 heartbeater.cc:503] Master 127.31.12.126:38891 was elected leader, sending a full tablet report...
I20260924 15:21:23.416554 15455 tablet_bootstrap.cc:492] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da: No bootstrap required, opened a new log
I20260924 15:21:23.416659 15455 ts_tablet_manager.cc:1397] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 15:21:23.418118 15455 raft_consensus.cc:348] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } }
I20260924 15:21:23.418246 15455 raft_consensus.cc:374] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:21:23.418267 15455 raft_consensus.cc:729] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c339766ca354ae88b148dc07f93b0da, State: Initialized, Role: FOLLOWER
I20260924 15:21:23.418385 15455 consensus_queue.cc:260] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } }
I20260924 15:21:23.418654 15455 ts_tablet_manager.cc:1428] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:21:23.418704 15437 heartbeater.cc:503] Master 127.31.12.126:38891 was elected leader, sending a full tablet report...
W20260924 15:21:23.455695 15300 tablet.cc:2369] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:21:23.515884 15457 raft_consensus.cc:482] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:21:23.516031 15457 raft_consensus.cc:504] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } }
I20260924 15:21:23.516484 15457 leader_election.cc:288] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dde3faf177d4fcbb16905769398d7fd (127.31.12.66:33753), 0c339766ca354ae88b148dc07f93b0da (127.31.12.67:33089)
I20260924 15:21:23.520305 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:23.516777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:53111 (local address 127.31.12.66:33753)
0924 15:21:23.516826 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:23.516829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:23.516840 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:21:23.516905 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:23.516906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:23.516915 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:23.516977 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:23.517162 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.517936 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.519329 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.519906 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.519984 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:23.519992 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:23.520197 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 15:21:23.520234 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 15:21:23.520305 15450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:23.516895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:47641 (local address 127.31.12.67:33089)
0924 15:21:23.516978 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:23.516981 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:23.517081 (+ 100us) server_negotiation.cc:407] Connection header received
0924 15:21:23.517157 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:23.517158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:23.517167 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:23.517239 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:23.517416 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.518090 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.519336 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.519906 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.519984 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:23.519992 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:23.520197 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 15:21:23.520234 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 15:21:23.520319 15162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:23.516702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:33753 (local address 127.31.12.65:53111)
0924 15:21:23.516756 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:23.516772 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:23.516843 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:23.516997 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:23.516998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:23.517000 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:23.517087 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:23.517089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.518019 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.518020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:23.519221 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:23.519224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.520007 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.520008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:23.520111 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:23.520113 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:21:23.520173 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 15:21:23.520223 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":25}
I20260924 15:21:23.520325 15460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:23.516818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:33089 (local address 127.31.12.65:47641)
0924 15:21:23.517003 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:23.517014 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:23.517079 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:23.517267 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:23.517268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:23.517269 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:23.517334 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:23.517335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.518165 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.518166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:23.519255 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:23.519257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.520007 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.520008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:23.520109 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:23.520113 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:21:23.520173 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 15:21:23.520219 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":4,"thread_start_us":109,"threads_started":1}
I20260924 15:21:23.520740 15253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee22ddfe7d784fb38e89a1cd073b0494" candidate_uuid: "c8795445a8444291aaf5234a6f1ca235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dde3faf177d4fcbb16905769398d7fd" is_pre_election: true
I20260924 15:21:23.520901 15253 raft_consensus.cc:2454] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8795445a8444291aaf5234a6f1ca235 in term 0.
I20260924 15:21:23.521080 15390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee22ddfe7d784fb38e89a1cd073b0494" candidate_uuid: "c8795445a8444291aaf5234a6f1ca235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c339766ca354ae88b148dc07f93b0da" is_pre_election: true
I20260924 15:21:23.521132 15046 leader_election.cc:302] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dde3faf177d4fcbb16905769398d7fd, c8795445a8444291aaf5234a6f1ca235; no voters:
I20260924 15:21:23.521204 15390 raft_consensus.cc:2454] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8795445a8444291aaf5234a6f1ca235 in term 0.
I20260924 15:21:23.521345 15457 raft_consensus.cc:2781] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:21:23.521427 15457 raft_consensus.cc:482] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:21:23.521476 15457 raft_consensus.cc:3037] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:21:23.522174 15457 raft_consensus.cc:504] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } }
I20260924 15:21:23.522315 15457 leader_election.cc:288] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [CANDIDATE]: Term 1 election: Requested vote from peers 0dde3faf177d4fcbb16905769398d7fd (127.31.12.66:33753), 0c339766ca354ae88b148dc07f93b0da (127.31.12.67:33089)
I20260924 15:21:23.522466 15253 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee22ddfe7d784fb38e89a1cd073b0494" candidate_uuid: "c8795445a8444291aaf5234a6f1ca235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dde3faf177d4fcbb16905769398d7fd"
I20260924 15:21:23.522495 15390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee22ddfe7d784fb38e89a1cd073b0494" candidate_uuid: "c8795445a8444291aaf5234a6f1ca235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c339766ca354ae88b148dc07f93b0da"
I20260924 15:21:23.522552 15253 raft_consensus.cc:3037] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:21:23.522584 15390 raft_consensus.cc:3037] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:21:23.523272 15253 raft_consensus.cc:2454] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8795445a8444291aaf5234a6f1ca235 in term 1.
I20260924 15:21:23.523334 15390 raft_consensus.cc:2454] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8795445a8444291aaf5234a6f1ca235 in term 1.
I20260924 15:21:23.523514 15046 leader_election.cc:302] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c339766ca354ae88b148dc07f93b0da, c8795445a8444291aaf5234a6f1ca235; no voters:
I20260924 15:21:23.523607 15457 raft_consensus.cc:2781] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:21:23.523766 15457 raft_consensus.cc:686] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [term 1 LEADER]: Becoming Leader. State: Replica: c8795445a8444291aaf5234a6f1ca235, State: Running, Role: LEADER
I20260924 15:21:23.523873 15457 consensus_queue.cc:237] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } }
I20260924 15:21:23.524704 14965 catalog_manager.cc:5495] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8795445a8444291aaf5234a6f1ca235 (127.31.12.65). New cstate: current_term: 1 leader_uuid: "c8795445a8444291aaf5234a6f1ca235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } health_report { overall_health: UNKNOWN } } }
I20260924 15:21:23.551805 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:23.549659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43790 (local address 127.31.12.65:42095)
0924 15:21:23.549706 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:23.549708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:23.549725 (+ 17us) server_negotiation.cc:407] Connection header received
0924 15:21:23.549787 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:23.549788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:23.549805 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:23.549866 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:21:23.549989 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.550450 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.551156 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:23.551333 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:23.551381 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:23.551513 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:21:23.551621 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:21:23.551680 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:23.551714 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 15:21:23.551731 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:21:23.553609 15390 raft_consensus.cc:1261] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da [term 1 FOLLOWER]: Refusing update from remote peer c8795445a8444291aaf5234a6f1ca235: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:21:23.553653 15253 raft_consensus.cc:1261] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd [term 1 FOLLOWER]: Refusing update from remote peer c8795445a8444291aaf5234a6f1ca235: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:21:23.553958 15457 consensus_queue.cc:1035] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:21:23.554096 15462 consensus_queue.cc:1035] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:21:23.556981 15467 mvcc.cc:205] Tried to move back new op lower bound from 7332918409432944640 to 7332918409314467840. Current Snapshot: MvccSnapshot[applied={T|T < 7332918409432944640}]
I20260924 15:21:23.558670 14964 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:21:23.558773 14964 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:21:23.559559 14964 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960: Sending DeleteTablet for 3 replicas of tablet ee22ddfe7d784fb38e89a1cd073b0494
I20260924 15:21:23.559867 15093 tablet_service.cc:1514] Processing DeleteTablet for tablet ee22ddfe7d784fb38e89a1cd073b0494 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:21:23 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:43784
I20260924 15:21:23.559952 15232 tablet_service.cc:1514] Processing DeleteTablet for tablet ee22ddfe7d784fb38e89a1cd073b0494 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:21:23 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:58442
I20260924 15:21:23.560007 15368 tablet_service.cc:1514] Processing DeleteTablet for tablet ee22ddfe7d784fb38e89a1cd073b0494 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:21:23 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:49942
I20260924 15:21:23.560197 15474 tablet_replica.cc:330] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235: stopping tablet replica
I20260924 15:21:23.560295 15474 raft_consensus.cc:2229] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:21:23.560329 15476 tablet_replica.cc:330] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da: stopping tablet replica
I20260924 15:21:23.560349 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 15029
I20260924 15:21:23.560469 15476 raft_consensus.cc:2229] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:21:23.560614 15476 raft_consensus.cc:2258] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:21:23.561190 15476 ts_tablet_manager.cc:1905] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:21:23.562088 15475 tablet_replica.cc:330] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd: stopping tablet replica
I20260924 15:21:23.562209 15475 raft_consensus.cc:2229] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:21:23.562335 15475 raft_consensus.cc:2258] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:21:23.563180 15476 ts_tablet_manager.cc:1918] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:21:23.563261 15476 log.cc:1196] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/ee22ddfe7d784fb38e89a1cd073b0494
I20260924 15:21:23.563534 15476 ts_tablet_manager.cc:1939] T ee22ddfe7d784fb38e89a1cd073b0494 P 0c339766ca354ae88b148dc07f93b0da: Deleting consensus metadata
I20260924 15:21:23.563915 14950 catalog_manager.cc:4841] TS 0c339766ca354ae88b148dc07f93b0da (127.31.12.67:33089): tablet ee22ddfe7d784fb38e89a1cd073b0494 (table test-table [id=e4a8a6de18d940fd89e85b5033958a54]) successfully deleted
I20260924 15:21:23.565747 15475 ts_tablet_manager.cc:1905] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 15:21:23.566474 14952 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.12.65:42095 (error 108)
I20260924 15:21:23.566751 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 15165
I20260924 15:21:23.566991 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:23.566875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:42095 (local address 127.0.0.1:43796)
0924 15:21:23.566920 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:23.566952 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.65:42095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 15:21:23.567088 14952 catalog_manager.cc:4584] TS c8795445a8444291aaf5234a6f1ca235 (127.31.12.65:42095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee22ddfe7d784fb38e89a1cd073b0494: Network error: Client connection negotiation failed: client connection to 127.31.12.65:42095: connect: Connection refused (error 111)
I20260924 15:21:23.572784 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 15305
I20260924 15:21:23.572914 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:23.572775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:33753 (local address 127.0.0.1:58452)
0924 15:21:23.572849 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:23.572877 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.66:33753: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 15:21:23.577175 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 14932
I20260924 15:21:43.583091 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:38891
--webserver_interface=127.31.12.126
--webserver_port=34303
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:38891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:21:43.666198 15478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:43.666388 15478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:43.666428 15478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:43.667959 15478 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 15:21:43.668028 15478 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 15:21:43.668049 15478 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 15:21:43.668064 15478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:21:43.668087 15478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:43.668111 15478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:43.668174 15478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:21:43.668222 15478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:21:43.670588 15478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:38891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:38891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=34303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:43.670986 15478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:43.671248 15478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:43.680871 15485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:43.680877 15484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:43.680897 15487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:43.681051 15478 server_base.cc:1034] running on GCE node
I20260924 15:21:43.681489 15478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:43.681835 15478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:43.682981 15478 hybrid_clock.cc:648] HybridClock initialized: now 1790263303682967 us; error 29 us; skew 500 ppm
Sep 24 15:21:43 dist-test-slave-zwbb krb5kdc[14919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263303, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:43.685635 15478 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:21:43.686174 15478 webserver.cc:460] Webserver started at http://127.31.12.126:34303/ using document root <none> and password file <none>
I20260924 15:21:43.686439 15478 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:43.686498 15478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:43.687929 15478 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:21:43.688598 15494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:43.688778 15478 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:21:43.688870 15478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "6b543a3dbe1a4175ba59673d0823c960"
format_stamp: "Formatted at 2026-09-24 15:21:22 on dist-test-slave-zwbb"
server_key: "807003b2459e6e8fe525a8d962a4d694"
server_key_iv: "62d2f80a3ff272e713562caa880432a9"
server_key_version: "encryptionkey@0"
I20260924 15:21:43.689276 15478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:43.700191 15478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:43.702143 15478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:43.702288 15478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:43.706338 15478 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:38891
I20260924 15:21:43.706391 15548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:38891 every 8 connection(s)
I20260924 15:21:43.706782 15478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:21:43.708273 15549 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:21:43.708559 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 15478
I20260924 15:21:43.708848 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:42095
--local_ip_for_outbound_sockets=127.31.12.65
--tserver_master_addrs=127.31.12.126:38891
--webserver_port=34099
--webserver_interface=127.31.12.65
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:21:43.709064 15549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960: Bootstrap starting.
I20260924 15:21:43.718186 15549 log.cc:824] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:43.720088 15549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:21:43.720396 15549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960: Bootstrap complete.
I20260924 15:21:43.722715 15549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b543a3dbe1a4175ba59673d0823c960" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38891 } }
I20260924 15:21:43.722961 15549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b543a3dbe1a4175ba59673d0823c960, State: Initialized, Role: FOLLOWER
I20260924 15:21:43.723170 15549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b543a3dbe1a4175ba59673d0823c960" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38891 } }
I20260924 15:21:43.723269 15549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:21:43.723328 15549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:21:43.723377 15549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 1 FOLLOWER]: Advancing to term 2
I20260924 15:21:43.724376 15549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b543a3dbe1a4175ba59673d0823c960" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38891 } }
I20260924 15:21:43.724493 15549 leader_election.cc:302] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b543a3dbe1a4175ba59673d0823c960; no voters:
I20260924 15:21:43.724696 15549 leader_election.cc:288] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 15:21:43.724730 15553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 15:21:43.724866 15553 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 2 LEADER]: Becoming Leader. State: Replica: 6b543a3dbe1a4175ba59673d0823c960, State: Running, Role: LEADER
I20260924 15:21:43.724992 15549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:21:43.725024 15553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b543a3dbe1a4175ba59673d0823c960" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38891 } }
I20260924 15:21:43.725330 15553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b543a3dbe1a4175ba59673d0823c960. Latest consensus state: current_term: 2 leader_uuid: "6b543a3dbe1a4175ba59673d0823c960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b543a3dbe1a4175ba59673d0823c960" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38891 } } }
I20260924 15:21:43.725337 15554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6b543a3dbe1a4175ba59673d0823c960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b543a3dbe1a4175ba59673d0823c960" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38891 } } }
I20260924 15:21:43.725548 15553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [sys.catalog]: This master's current role is: LEADER
I20260924 15:21:43.725560 15554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [sys.catalog]: This master's current role is: LEADER
I20260924 15:21:43.725806 15562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:21:43.726871 15562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:21:43.726990 15562 catalog_manager.cc:1259] Loaded cluster ID: e475393132694bdca06419820774abaa
I20260924 15:21:43.727077 15562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:21:43.728053 15562 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:21:43.728245 15562 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960: Loaded TSK: 0
I20260924 15:21:43.738948 15562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960: Generated new TSK 1
I20260924 15:21:43.739074 15562 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 15:21:43.799109 15551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:43.799337 15551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:43.799378 15551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:43.800515 15551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:43.800580 15551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:43.800624 15551 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:21:43.802150 15551 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:42095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=34099
--tserver_master_addrs=127.31.12.126:38891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:43.802397 15551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:43.802615 15551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:43.807505 15576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:43.807509 15577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:43.807802 15579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:43.808013 15551 server_base.cc:1034] running on GCE node
I20260924 15:21:43.808210 15551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:43.808461 15551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:43.809631 15551 hybrid_clock.cc:648] HybridClock initialized: now 1790263303809617 us; error 31 us; skew 500 ppm
Sep 24 15:21:43 dist-test-slave-zwbb krb5kdc[14919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263303, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:43.812217 15551 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:21:43.812688 15551 webserver.cc:460] Webserver started at http://127.31.12.65:34099/ using document root <none> and password file <none>
I20260924 15:21:43.812906 15551 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:43.812989 15551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:43.814466 15551 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:21:43.815161 15586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:43.815346 15551 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:21:43.815442 15551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "c8795445a8444291aaf5234a6f1ca235"
format_stamp: "Formatted at 2026-09-24 15:21:23 on dist-test-slave-zwbb"
server_key: "dfd4ee048e2f19e023497f537bd3292a"
server_key_iv: "875d18effd49d0509d57fab07ce9bc37"
server_key_version: "encryptionkey@0"
I20260924 15:21:43.815769 15551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:43.833669 15551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:43.835892 15551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:43.836076 15551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:43.836313 15551 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:21:43.836774 15593 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 15:21:43.838761 15551 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 15:21:43.838829 15551 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 15:21:43.838864 15551 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 15:21:43.839485 15551 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 15:21:43.839552 15551 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 15:21:43.839617 15593 tablet_bootstrap.cc:492] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235: Bootstrap starting.
I20260924 15:21:43.846457 15551 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:42095
I20260924 15:21:43.846506 15702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:42095 every 8 connection(s)
I20260924 15:21:43.846885 15551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:21:43.850379 15593 log.cc:824] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:43.852913 15593 tablet_bootstrap.cc:492] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:21:43.853235 15593 tablet_bootstrap.cc:492] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235: Bootstrap complete.
I20260924 15:21:43.854599 15593 ts_tablet_manager.cc:1397] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.006s
I20260924 15:21:43.855329 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 15551
I20260924 15:21:43.855648 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:33753
--local_ip_for_outbound_sockets=127.31.12.66
--tserver_master_addrs=127.31.12.126:38891
--webserver_port=38315
--webserver_interface=127.31.12.66
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:21:43.855930 15593 raft_consensus.cc:348] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } }
I20260924 15:21:43.856205 15593 raft_consensus.cc:729] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8795445a8444291aaf5234a6f1ca235, State: Initialized, Role: FOLLOWER
I20260924 15:21:43.856370 15593 consensus_queue.cc:260] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } }
I20260924 15:21:43.856611 15593 ts_tablet_manager.cc:1428] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
Sep 24 15:21:43 dist-test-slave-zwbb krb5kdc[14919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263303, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:21:43.865561 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:43.847600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.65:32987)
0924 15:21:43.853174 (+ 5574us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:43.853230 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:43.854054 (+ 824us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:43.854493 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:43.854496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:43.854843 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:43.855042 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:43.855047 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:43.856937 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:43.856938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:43.857962 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:43.857965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:43.858562 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:43.858563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:43.858699 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:43.859260 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:43.859271 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:43.861052 (+ 1781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:43.863629 (+ 2577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:43.863637 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:43.863639 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:43.863915 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:43.864237 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:43.864238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:43.864238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:43.864367 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:43.864876 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:43.864883 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:43.865037 (+ 154us) client_negotiation.cc:769] Sending connection context
0924 15:21:43.865148 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 15:21:43.865299 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5517,"thread_start_us":101,"threads_started":1}
I20260924 15:21:43.865674 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:43.847658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:32987 (local address 127.31.12.126:38891)
0924 15:21:43.853535 (+ 5877us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:43.853544 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:43.853587 (+ 43us) server_negotiation.cc:407] Connection header received
0924 15:21:43.854282 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:43.854287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:43.854331 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:43.854464 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:43.855160 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:43.856630 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:43.858056 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:43.858461 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:43.858589 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:43.861207 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:43.861212 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:43.861212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:43.861226 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:43.863533 (+ 2307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:43.864008 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:43.864012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:43.864012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:43.864153 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:43.864453 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:43.864454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:43.864455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:43.864776 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:43.864892 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:43.865234 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 15:21:43.865391 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5825,"thread_start_us":72,"threads_started":1}
I20260924 15:21:43.866770 15703 heartbeater.cc:347] Connected to a master server at 127.31.12.126:38891
I20260924 15:21:43.866875 15703 heartbeater.cc:464] Registering TS with master...
I20260924 15:21:43.867087 15703 heartbeater.cc:511] Master 127.31.12.126:38891 requested a full tablet report, sending...
I20260924 15:21:43.867867 15512 ts_manager.cc:194] Registered new tserver with Master: c8795445a8444291aaf5234a6f1ca235 (127.31.12.65:42095)
I20260924 15:21:43.869084 15512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:32987
I20260924 15:21:43.870472 15703 heartbeater.cc:503] Master 127.31.12.126:38891 was elected leader, sending a full tablet report...
I20260924 15:21:43.874270 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:43.869480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:42095 (local address 127.0.0.1:52806)
0924 15:21:43.869908 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:43.869939 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:43.870000 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:43.870463 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:43.870466 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:43.870468 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:43.870524 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:43.870526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:43.871756 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:43.871757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:43.873079 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:43.873082 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:43.873970 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:43.873971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:43.874064 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:43.874079 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 15:21:43.874147 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 15:21:43.874195 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":82,"threads_started":1}
I20260924 15:21:43.874293 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:43.869591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52806 (local address 127.31.12.65:42095)
0924 15:21:43.869755 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:43.869760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:43.870071 (+ 311us) server_negotiation.cc:407] Connection header received
0924 15:21:43.870106 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:43.870108 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:43.870327 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:43.870396 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:43.870592 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:43.871668 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:43.873177 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:43.873875 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:43.873947 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:43.873997 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:43.874175 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 15:21:43.874214 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260924 15:21:43.874838 15622 tablet_service.cc:1514] Processing DeleteTablet for tablet ee22ddfe7d784fb38e89a1cd073b0494 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:21:23 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:52806
I20260924 15:21:43.875536 15712 tablet_replica.cc:330] stopping tablet replica
I20260924 15:21:43.875656 15712 raft_consensus.cc:2229] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:21:43.875747 15712 raft_consensus.cc:2258] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:21:43.876057 15712 ts_tablet_manager.cc:1905] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:21:43.878165 15712 ts_tablet_manager.cc:1918] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:21:43.878254 15712 log.cc:1196] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/wal/wals/ee22ddfe7d784fb38e89a1cd073b0494
I20260924 15:21:43.878487 15712 ts_tablet_manager.cc:1939] T ee22ddfe7d784fb38e89a1cd073b0494 P c8795445a8444291aaf5234a6f1ca235: Deleting consensus metadata
I20260924 15:21:43.878948 15499 catalog_manager.cc:4841] TS c8795445a8444291aaf5234a6f1ca235 (127.31.12.65:42095): tablet ee22ddfe7d784fb38e89a1cd073b0494 (table test-table [id=e4a8a6de18d940fd89e85b5033958a54]) successfully deleted
W20260924 15:21:43.942692 15709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:43.942931 15709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:43.943037 15709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:43.944176 15709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:43.944212 15709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:43.944238 15709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:21:43.945772 15709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:33753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=38315
--tserver_master_addrs=127.31.12.126:38891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:43.945964 15709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:43.946138 15709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:43.950954 15718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:43.950947 15717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:43.950946 15720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:43.951146 15709 server_base.cc:1034] running on GCE node
I20260924 15:21:43.951366 15709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:43.951608 15709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:43.952790 15709 hybrid_clock.cc:648] HybridClock initialized: now 1790263303952778 us; error 29 us; skew 500 ppm
Sep 24 15:21:43 dist-test-slave-zwbb krb5kdc[14919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263303, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:43.955125 15709 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:21:43.955596 15709 webserver.cc:460] Webserver started at http://127.31.12.66:38315/ using document root <none> and password file <none>
I20260924 15:21:43.955849 15709 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:43.955933 15709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:43.957221 15709 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:21:43.957974 15727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:43.958160 15709 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:43.958261 15709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "0dde3faf177d4fcbb16905769398d7fd"
format_stamp: "Formatted at 2026-09-24 15:21:23 on dist-test-slave-zwbb"
server_key: "e9593341322ff8e1f86dd06834ebd980"
server_key_iv: "8dd3239116d4fbc18db82283e5b568c3"
server_key_version: "encryptionkey@0"
I20260924 15:21:43.958580 15709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:43.967808 15709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:43.969501 15709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:43.969668 15709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:43.969887 15709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:21:43.970304 15734 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260924 15:21:43.971563 15734 ts_tablet_manager.cc:1852] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260924 15:21:43.971805 15734 ts_tablet_manager.cc:1905] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:21:43.973912 15734 ts_tablet_manager.cc:1918] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:21:43.973982 15734 log.cc:1196] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/ee22ddfe7d784fb38e89a1cd073b0494
I20260924 15:21:43.974207 15734 ts_tablet_manager.cc:1939] T ee22ddfe7d784fb38e89a1cd073b0494 P 0dde3faf177d4fcbb16905769398d7fd: Deleting consensus metadata
I20260924 15:21:43.974350 15709 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260924 15:21:43.974411 15709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260924 15:21:43.974453 15709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:21:43.974480 15709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:43.980461 15709 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:33753
I20260924 15:21:43.980520 15844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:33753 every 8 connection(s)
I20260924 15:21:43.980805 15709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:21:43.982223 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 15709
I20260924 15:21:43.982522 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:33089
--local_ip_for_outbound_sockets=127.31.12.67
--tserver_master_addrs=127.31.12.126:38891
--webserver_port=46271
--webserver_interface=127.31.12.67
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 15:21:43 dist-test-slave-zwbb krb5kdc[14919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263303, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:21:43.991302 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:43.981419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.66:56193)
0924 15:21:43.981675 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:43.981709 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:43.982157 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:43.982450 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:43.982454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:43.982795 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:43.982986 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:43.982990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:43.983962 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:43.983963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:43.985303 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:43.985306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:43.985748 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:43.985749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:43.985898 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:43.986424 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:43.986434 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:43.988001 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:43.989899 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:43.989905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:43.989907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:43.990123 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:43.990341 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:43.990342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:43.990342 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:43.990438 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:43.990734 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:43.990738 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:43.990916 (+ 178us) client_negotiation.cc:769] Sending connection context
0924 15:21:43.991022 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 15:21:43.991147 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":174,"threads_started":1}
I20260924 15:21:43.991302 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:43.981474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:56193 (local address 127.31.12.126:38891)
0924 15:21:43.981528 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:43.981531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:43.981773 (+ 242us) server_negotiation.cc:407] Connection header received
0924 15:21:43.982259 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:43.982261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:43.982279 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:43.982356 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:43.983084 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:43.983853 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:43.985391 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:43.985661 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:43.985719 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:43.988095 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:43.988097 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:43.988098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:43.988106 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:43.989822 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:43.990201 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:43.990202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:43.990203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:43.990269 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:43.990511 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:43.990512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:43.990512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:43.990664 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:43.990741 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:43.991070 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 15:21:43.991170 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:21:43.991809 15845 heartbeater.cc:347] Connected to a master server at 127.31.12.126:38891
I20260924 15:21:43.991916 15845 heartbeater.cc:464] Registering TS with master...
I20260924 15:21:43.992115 15845 heartbeater.cc:511] Master 127.31.12.126:38891 requested a full tablet report, sending...
I20260924 15:21:43.992573 15512 ts_manager.cc:194] Registered new tserver with Master: 0dde3faf177d4fcbb16905769398d7fd (127.31.12.66:33753)
I20260924 15:21:43.992942 15512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:56193
W20260924 15:21:44.065855 15848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:21:44.066057 15848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:21:44.066080 15848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:21:44.067209 15848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:21:44.067248 15848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:21:44.067305 15848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:21:44.068724 15848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:33089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=46271
--tserver_master_addrs=127.31.12.126:38891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:21:44.068948 15848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:21:44.069173 15848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:21:44.074100 15854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:21:44.074093 15853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:44.074216 15848 server_base.cc:1034] running on GCE node
W20260924 15:21:44.074092 15856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:21:44.074477 15848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:21:44.074723 15848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:21:44.075861 15848 hybrid_clock.cc:648] HybridClock initialized: now 1790263304075852 us; error 28 us; skew 500 ppm
Sep 24 15:21:44 dist-test-slave-zwbb krb5kdc[14919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263304, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:21:44.078243 15848 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:21:44.078615 15848 webserver.cc:460] Webserver started at http://127.31.12.67:46271/ using document root <none> and password file <none>
I20260924 15:21:44.078787 15848 fs_manager.cc:359] Metadata directory not provided
I20260924 15:21:44.078838 15848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:21:44.080076 15848 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:21:44.080612 15863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:44.080760 15848 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:21:44.080840 15848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "0c339766ca354ae88b148dc07f93b0da"
format_stamp: "Formatted at 2026-09-24 15:21:23 on dist-test-slave-zwbb"
server_key: "1eabcafc97b76446cf215ad1cdd19a3f"
server_key_iv: "396a3591f02a627a588eb0b1a2729b76"
server_key_version: "encryptionkey@0"
I20260924 15:21:44.081136 15848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:21:44.101274 15848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:21:44.103152 15848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:21:44.103257 15848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:21:44.103430 15848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:21:44.103752 15848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:21:44.103786 15848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:44.103808 15848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:21:44.103859 15848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:44.109479 15848 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:33089
I20260924 15:21:44.109565 15980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:33089 every 8 connection(s)
I20260924 15:21:44.109915 15848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:21:44 dist-test-slave-zwbb krb5kdc[14919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263304, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:21:44.118460 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 15848
I20260924 15:21:44.119751 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:44.110635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.67:41715)
0924 15:21:44.110811 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:44.110851 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:44.111355 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:44.111629 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:44.111633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:44.112202 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:21:44.112390 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:44.112394 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.113123 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.113124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:44.113997 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:44.114000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.114408 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.114409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:44.114563 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:44.115147 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:21:44.115158 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:21:44.116756 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:21:44.118254 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:44.118262 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:44.118264 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:44.118482 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:44.118776 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:21:44.118777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:21:44.118777 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:21:44.118912 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:21:44.119238 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:21:44.119245 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:21:44.119391 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 15:21:44.119475 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 15:21:44.119595 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":91,"threads_started":1}
I20260924 15:21:44.119750 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:44.110660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:41715 (local address 127.31.12.126:38891)
0924 15:21:44.110729 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:44.110746 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:44.110931 (+ 185us) server_negotiation.cc:407] Connection header received
0924 15:21:44.111470 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:44.111472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:44.111486 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:44.111552 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:44.112504 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.113023 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.114111 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.114312 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.114403 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:44.116856 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:44.116857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:44.116857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:44.116865 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:44.118153 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:44.118615 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:44.118616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:44.118617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:44.118678 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:44.119011 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:44.119012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:44.119013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:44.119171 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:44.119242 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:44.119534 (+ 292us) server_negotiation.cc:299] Negotiation successful
0924 15:21:44.119635 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 15:21:44.120308 15981 heartbeater.cc:347] Connected to a master server at 127.31.12.126:38891
I20260924 15:21:44.120419 15981 heartbeater.cc:464] Registering TS with master...
I20260924 15:21:44.120597 15981 heartbeater.cc:511] Master 127.31.12.126:38891 requested a full tablet report, sending...
I20260924 15:21:44.121012 15512 ts_manager.cc:194] Registered new tserver with Master: 0c339766ca354ae88b148dc07f93b0da (127.31.12.67:33089)
I20260924 15:21:44.121369 15512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:41715
I20260924 15:21:44.125442 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:44.118856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58594 (local address 127.31.12.126:38891)
0924 15:21:44.118966 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:44.118968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:44.119104 (+ 136us) server_negotiation.cc:407] Connection header received
0924 15:21:44.119160 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:44.119160 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:44.119170 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:44.119242 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:44.120047 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.120623 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.121428 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.121657 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.121742 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:44.122906 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:44.122908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:44.122908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:44.122914 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:44.124278 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:44.124666 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:44.124667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:44.124667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:44.124721 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:44.124930 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:44.124930 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:44.124931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:44.125042 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:44.125231 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:44.125267 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 15:21:44.125341 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":62,"threads_started":1}
I20260924 15:21:44.126430 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:21:44.128784 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:44.126765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58598 (local address 127.31.12.126:38891)
0924 15:21:44.126824 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:44.126826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:44.126926 (+ 100us) server_negotiation.cc:407] Connection header received
0924 15:21:44.126959 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:44.126960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:44.126987 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:44.127043 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:21:44.127117 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.127538 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.128226 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.128396 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.128443 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:44.128584 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:21:44.128619 (+ 35us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 15:21:44.128724 (+ 105us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58598: token expired
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 15:21:44.128837 15985 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58598: token expired
W20260924 15:21:44.128940 15440 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.31.12.126:38891)
I20260924 15:21:44.134248 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:44.129135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58610 (local address 127.31.12.126:38891)
0924 15:21:44.129197 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:44.129198 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:44.129205 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:21:44.129220 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:44.129220 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:44.129226 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:44.129266 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:21:44.129769 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.130182 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.130803 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.130963 (+ 160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.131017 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:44.131987 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:21:44.131988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:21:44.131988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:21:44.132001 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:21:44.133305 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:44.133673 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:44.133674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:44.133674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:44.133720 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:21:44.133844 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:21:44.133844 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:21:44.133844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:21:44.133946 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:21:44.134023 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:44.134104 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 15:21:44.134158 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:21:44.135279 15512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:21:44.136869 15512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:21:44.139061 15622 tablet_service.cc:1467] Processing CreateTablet for tablet e789a8c0c13e49dcbf114c96d28096dc (DEFAULT_TABLE table=test-table [id=c8f2e2e4e2dc48328dc544ca29210e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:21:44.139258 15622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e789a8c0c13e49dcbf114c96d28096dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:21:44.140704 15593 tablet_bootstrap.cc:492] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235: Bootstrap starting.
I20260924 15:21:44.141186 15593 tablet_bootstrap.cc:654] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235: Neither blocks nor log segments found. Creating new log.
I20260924 15:21:44.141847 15593 tablet_bootstrap.cc:492] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235: No bootstrap required, opened a new log
I20260924 15:21:44.141892 15593 ts_tablet_manager.cc:1397] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 15:21:44.142025 15593 raft_consensus.cc:348] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } }
I20260924 15:21:44.142083 15593 raft_consensus.cc:374] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:21:44.142122 15593 raft_consensus.cc:729] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8795445a8444291aaf5234a6f1ca235, State: Initialized, Role: FOLLOWER
I20260924 15:21:44.142166 15593 consensus_queue.cc:260] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } }
I20260924 15:21:44.142251 15593 ts_tablet_manager.cc:1428] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 15:21:44.144198 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:44.138771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:33089 (local address 127.0.0.1:35978)
0924 15:21:44.138809 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:44.138821 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:44.138853 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:44.140945 (+ 2092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:44.140946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:44.140948 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:44.140988 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:44.140989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.142336 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.142337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:44.143128 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:44.143130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.143916 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.143917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:44.144012 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:44.144016 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:21:44.144073 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:21:44.144113 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 15:21:44.144389 15989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:44.138869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:33753 (local address 127.0.0.1:33664)
0924 15:21:44.141016 (+ 2147us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:44.141025 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:44.141099 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:44.141337 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:44.141338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:44.141340 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:44.141377 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:44.141378 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.142639 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.142640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:44.143460 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:44.143463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.144192 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.144192 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:44.144252 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:44.144255 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:21:44.144288 (+ 33us) client_negotiation.cc:240] Negotiation successful
0924 15:21:44.144331 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2134,"thread_start_us":96,"threads_started":1}
I20260924 15:21:44.144482 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:44.138857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35978 (local address 127.31.12.67:33089)
0924 15:21:44.140782 (+ 1925us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:44.140786 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:44.140796 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:21:44.140833 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:44.140835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:44.140850 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:44.140929 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:44.141076 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.142204 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.143212 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.143836 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.143891 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:44.143928 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:44.144357 (+ 429us) server_negotiation.cc:299] Negotiation successful
0924 15:21:44.144400 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1890,"thread_start_us":103,"threads_started":1}
I20260924 15:21:44.144905 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:44.138979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33664 (local address 127.31.12.66:33753)
0924 15:21:44.139203 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:44.139208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:44.141101 (+ 1893us) server_negotiation.cc:407] Connection header received
0924 15:21:44.141165 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:44.141168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:44.141184 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:44.141265 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:44.141470 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.142560 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.143530 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.144106 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.144506 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:44.144732 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:44.144752 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 15:21:44.144783 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":113,"threads_started":1}
I20260924 15:21:44.145188 15914 tablet_service.cc:1467] Processing CreateTablet for tablet e789a8c0c13e49dcbf114c96d28096dc (DEFAULT_TABLE table=test-table [id=c8f2e2e4e2dc48328dc544ca29210e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:21:44.145565 15914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e789a8c0c13e49dcbf114c96d28096dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:21:44.145546 15778 tablet_service.cc:1467] Processing CreateTablet for tablet e789a8c0c13e49dcbf114c96d28096dc (DEFAULT_TABLE table=test-table [id=c8f2e2e4e2dc48328dc544ca29210e94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:21:44.145823 15778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e789a8c0c13e49dcbf114c96d28096dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:21:44.147588 15734 tablet_bootstrap.cc:492] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd: Bootstrap starting.
I20260924 15:21:44.148406 15734 tablet_bootstrap.cc:654] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd: Neither blocks nor log segments found. Creating new log.
I20260924 15:21:44.148700 15992 tablet_bootstrap.cc:492] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da: Bootstrap starting.
I20260924 15:21:44.148746 15734 log.cc:824] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:44.149255 15992 tablet_bootstrap.cc:654] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da: Neither blocks nor log segments found. Creating new log.
I20260924 15:21:44.149510 15734 tablet_bootstrap.cc:492] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd: No bootstrap required, opened a new log
I20260924 15:21:44.149595 15734 ts_tablet_manager.cc:1397] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:21:44.149834 15992 log.cc:824] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da: Log is configured to *not* fsync() on all Append() calls
I20260924 15:21:44.150574 15992 tablet_bootstrap.cc:492] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da: No bootstrap required, opened a new log
I20260924 15:21:44.150628 15992 ts_tablet_manager.cc:1397] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 15:21:44.151306 15734 raft_consensus.cc:348] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } }
I20260924 15:21:44.151456 15734 raft_consensus.cc:374] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:21:44.151503 15734 raft_consensus.cc:729] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dde3faf177d4fcbb16905769398d7fd, State: Initialized, Role: FOLLOWER
I20260924 15:21:44.151628 15734 consensus_queue.cc:260] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } }
I20260924 15:21:44.151899 15734 ts_tablet_manager.cc:1428] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:21:44.151911 15845 heartbeater.cc:503] Master 127.31.12.126:38891 was elected leader, sending a full tablet report...
I20260924 15:21:44.152158 15992 raft_consensus.cc:348] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } }
I20260924 15:21:44.152303 15992 raft_consensus.cc:374] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:21:44.152383 15992 raft_consensus.cc:729] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c339766ca354ae88b148dc07f93b0da, State: Initialized, Role: FOLLOWER
I20260924 15:21:44.152504 15992 consensus_queue.cc:260] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } }
I20260924 15:21:44.152750 15992 ts_tablet_manager.cc:1428] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:21:44.152782 15981 heartbeater.cc:503] Master 127.31.12.126:38891 was elected leader, sending a full tablet report...
W20260924 15:21:44.231333 15846 tablet.cc:2369] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:21:44.354348 15704 tablet.cc:2369] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:21:44.360484 15982 tablet.cc:2369] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:21:44.512838 15996 raft_consensus.cc:482] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:21:44.512979 15996 raft_consensus.cc:504] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } }
I20260924 15:21:44.513360 15996 leader_election.cc:288] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dde3faf177d4fcbb16905769398d7fd (127.31.12.66:33753), c8795445a8444291aaf5234a6f1ca235 (127.31.12.65:42095)
I20260924 15:21:44.516130 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:44.513575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:33753 (local address 127.31.12.67:50731)
0924 15:21:44.513618 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:44.513630 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:44.513682 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:44.513848 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:44.513849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:44.513851 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:44.513937 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:44.513939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.514588 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.514589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:44.515344 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:44.515346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.515879 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.515880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:44.515958 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:44.515961 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:21:44.516018 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:21:44.516059 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21,"mutex_wait_us":1}
I20260924 15:21:44.516130 15990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:44.513658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:50731 (local address 127.31.12.66:33753)
0924 15:21:44.513709 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:44.513711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:44.513722 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:21:44.513755 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:44.513756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:44.513770 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:44.513831 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:44.514015 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.514503 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.515438 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.515795 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.515877 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:44.515886 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:44.516039 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 15:21:44.516073 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 15:21:44.516304 15998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:44.513720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:54893 (local address 127.31.12.65:42095)
0924 15:21:44.513844 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:44.513846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:44.513877 (+ 31us) server_negotiation.cc:407] Connection header received
0924 15:21:44.513958 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:44.513959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:44.513968 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:44.514065 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:21:44.514224 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.514692 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.515605 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.515958 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.516055 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:44.516063 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:44.516200 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 15:21:44.516240 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260924 15:21:44.516309 15997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:44.513657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:42095 (local address 127.31.12.67:54893)
0924 15:21:44.513818 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:44.513827 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:44.513878 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:44.514067 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:21:44.514068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:21:44.514069 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:21:44.514149 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:44.514151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.514775 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.514776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:44.515503 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:21:44.515505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.516058 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.516059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:21:44.516129 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:44.516131 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:21:44.516194 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 15:21:44.516232 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":22,"thread_start_us":108,"threads_started":1}
I20260924 15:21:44.516494 15798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e789a8c0c13e49dcbf114c96d28096dc" candidate_uuid: "0c339766ca354ae88b148dc07f93b0da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dde3faf177d4fcbb16905769398d7fd" is_pre_election: true
I20260924 15:21:44.516649 15798 raft_consensus.cc:2454] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c339766ca354ae88b148dc07f93b0da in term 0.
I20260924 15:21:44.516700 15655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e789a8c0c13e49dcbf114c96d28096dc" candidate_uuid: "0c339766ca354ae88b148dc07f93b0da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8795445a8444291aaf5234a6f1ca235" is_pre_election: true
I20260924 15:21:44.516873 15655 raft_consensus.cc:2454] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c339766ca354ae88b148dc07f93b0da in term 0.
I20260924 15:21:44.516889 15865 leader_election.cc:302] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c339766ca354ae88b148dc07f93b0da, 0dde3faf177d4fcbb16905769398d7fd; no voters:
I20260924 15:21:44.517024 15996 raft_consensus.cc:2781] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:21:44.517102 15996 raft_consensus.cc:482] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:21:44.517135 15996 raft_consensus.cc:3037] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:21:44.519045 15996 raft_consensus.cc:504] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } }
I20260924 15:21:44.519207 15996 leader_election.cc:288] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [CANDIDATE]: Term 1 election: Requested vote from peers 0dde3faf177d4fcbb16905769398d7fd (127.31.12.66:33753), c8795445a8444291aaf5234a6f1ca235 (127.31.12.65:42095)
I20260924 15:21:44.519449 15655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e789a8c0c13e49dcbf114c96d28096dc" candidate_uuid: "0c339766ca354ae88b148dc07f93b0da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8795445a8444291aaf5234a6f1ca235"
I20260924 15:21:44.519495 15798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e789a8c0c13e49dcbf114c96d28096dc" candidate_uuid: "0c339766ca354ae88b148dc07f93b0da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dde3faf177d4fcbb16905769398d7fd"
I20260924 15:21:44.519556 15655 raft_consensus.cc:3037] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:21:44.519569 15798 raft_consensus.cc:3037] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:21:44.520387 15655 raft_consensus.cc:2454] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c339766ca354ae88b148dc07f93b0da in term 1.
I20260924 15:21:44.520401 15798 raft_consensus.cc:2454] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c339766ca354ae88b148dc07f93b0da in term 1.
I20260924 15:21:44.520584 15867 leader_election.cc:302] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c339766ca354ae88b148dc07f93b0da, c8795445a8444291aaf5234a6f1ca235; no voters:
I20260924 15:21:44.520720 15996 raft_consensus.cc:2781] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:21:44.520886 15996 raft_consensus.cc:686] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [term 1 LEADER]: Becoming Leader. State: Replica: 0c339766ca354ae88b148dc07f93b0da, State: Running, Role: LEADER
I20260924 15:21:44.520969 15996 consensus_queue.cc:237] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } }
I20260924 15:21:44.521780 15512 catalog_manager.cc:5495] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c339766ca354ae88b148dc07f93b0da (127.31.12.67). New cstate: current_term: 1 leader_uuid: "0c339766ca354ae88b148dc07f93b0da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } health_report { overall_health: UNKNOWN } } }
I20260924 15:21:44.584146 15988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:21:44.581598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35986 (local address 127.31.12.67:33089)
0924 15:21:44.581678 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 15:21:44.581680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:21:44.581690 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:21:44.581771 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:21:44.581771 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:21:44.581794 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:21:44.581885 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:21:44.582024 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.582509 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.583272 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:21:44.583477 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:21:44.583567 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:21:44.583742 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:21:44.583888 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:21:44.583977 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 15:21:44.584019 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 15:21:44.584040 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:21:44.585990 15798 raft_consensus.cc:1261] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd [term 1 FOLLOWER]: Refusing update from remote peer 0c339766ca354ae88b148dc07f93b0da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:21:44.585988 15655 raft_consensus.cc:1261] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Refusing update from remote peer 0c339766ca354ae88b148dc07f93b0da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:21:44.586339 15996 consensus_queue.cc:1035] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:21:44.586508 15999 consensus_queue.cc:1035] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:21:44.589048 16003 mvcc.cc:205] Tried to move back new op lower bound from 7332918495581233152 to 7332918495318351872. Current Snapshot: MvccSnapshot[applied={T|T < 7332918495581233152}]
I20260924 15:21:44.589318 16004 mvcc.cc:205] Tried to move back new op lower bound from 7332918495581233152 to 7332918495318351872. Current Snapshot: MvccSnapshot[applied={T|T < 7332918495581233152 or (T in {7332918495581233152})}]
I20260924 15:21:44.590899 15511 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:21:44.590991 15511 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:21:44.591710 15511 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960: Sending DeleteTablet for 3 replicas of tablet e789a8c0c13e49dcbf114c96d28096dc
I20260924 15:21:44.592017 15622 tablet_service.cc:1514] Processing DeleteTablet for tablet e789a8c0c13e49dcbf114c96d28096dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:21:44 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:52806
I20260924 15:21:44.592024 15778 tablet_service.cc:1514] Processing DeleteTablet for tablet e789a8c0c13e49dcbf114c96d28096dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:21:44 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:33664
I20260924 15:21:44.592053 15914 tablet_service.cc:1514] Processing DeleteTablet for tablet e789a8c0c13e49dcbf114c96d28096dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:21:44 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:35978
I20260924 15:21:44.592263 16011 tablet_replica.cc:330] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235: stopping tablet replica
I20260924 15:21:44.592315 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 15551
I20260924 15:21:44.592322 16011 raft_consensus.cc:2229] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:21:44.592464 16013 tablet_replica.cc:330] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da: stopping tablet replica
I20260924 15:21:44.592638 16013 raft_consensus.cc:2229] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:21:44.592371 16011 raft_consensus.cc:2258] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:21:44.592873 16013 raft_consensus.cc:2258] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:21:44.593197 16012 tablet_replica.cc:330] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd: stopping tablet replica
I20260924 15:21:44.593343 16012 raft_consensus.cc:2229] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:21:44.593504 16012 raft_consensus.cc:2258] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:21:44.593583 16013 ts_tablet_manager.cc:1905] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:21:44.593974 16012 ts_tablet_manager.cc:1905] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:21:44.595539 16013 ts_tablet_manager.cc:1918] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:21:44.595646 16013 log.cc:1196] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/e789a8c0c13e49dcbf114c96d28096dc
I20260924 15:21:44.595896 16012 ts_tablet_manager.cc:1918] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:21:44.595984 16013 ts_tablet_manager.cc:1939] T e789a8c0c13e49dcbf114c96d28096dc P 0c339766ca354ae88b148dc07f93b0da: Deleting consensus metadata
I20260924 15:21:44.596017 16012 log.cc:1196] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/e789a8c0c13e49dcbf114c96d28096dc
I20260924 15:21:44.596276 16012 ts_tablet_manager.cc:1939] T e789a8c0c13e49dcbf114c96d28096dc P 0dde3faf177d4fcbb16905769398d7fd: Deleting consensus metadata
I20260924 15:21:44.596443 15497 catalog_manager.cc:4841] TS 0c339766ca354ae88b148dc07f93b0da (127.31.12.67:33089): tablet e789a8c0c13e49dcbf114c96d28096dc (table test-table [id=c8f2e2e4e2dc48328dc544ca29210e94]) successfully deleted
I20260924 15:21:44.596696 15497 catalog_manager.cc:4841] TS 0dde3faf177d4fcbb16905769398d7fd (127.31.12.66:33753): tablet e789a8c0c13e49dcbf114c96d28096dc (table test-table [id=c8f2e2e4e2dc48328dc544ca29210e94]) successfully deleted
W20260924 15:21:44.599517 15499 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.12.65:42095 (error 108)
W20260924 15:21:44.600139 15989 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:21:44.599917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:42095 (local address 127.0.0.1:52812)
0924 15:21:44.599982 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0924 15:21:44.599994 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:21:44.600049 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:21:44.600093 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.65:42095: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 15:21:44.600225 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 15709
W20260924 15:21:44.600404 15499 catalog_manager.cc:4584] TS c8795445a8444291aaf5234a6f1ca235 (127.31.12.65:42095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e789a8c0c13e49dcbf114c96d28096dc: Network error: Client connection negotiation failed: client connection to 127.31.12.65:42095: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260924 15:21:44.605926 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 15848
I20260924 15:21:44.610500 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 15478
I20260924 15:22:04.615069 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:38891
--webserver_interface=127.31.12.126
--webserver_port=34303
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:38891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:04.694859 16015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:04.695065 16015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:04.695087 16015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:04.696225 16015 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 15:22:04.696264 16015 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 15:22:04.696277 16015 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 15:22:04.696321 16015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:04.696341 16015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:04.696364 16015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:04.696393 16015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:04.696436 16015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:04.698038 16015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:38891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:38891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=34303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:04.698282 16015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:04.698518 16015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:04.703706 16021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:04.703737 16020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:04.703706 16023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:04.703769 16015 server_base.cc:1034] running on GCE node
I20260924 15:22:04.704149 16015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:04.704401 16015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:04.705566 16015 hybrid_clock.cc:648] HybridClock initialized: now 1790263324705554 us; error 40 us; skew 500 ppm
Sep 24 15:22:04 dist-test-slave-zwbb krb5kdc[14919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263324, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:04.708367 16015 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:22:04.708935 16015 webserver.cc:460] Webserver started at http://127.31.12.126:34303/ using document root <none> and password file <none>
I20260924 15:22:04.709287 16015 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:04.709369 16015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:04.711280 16015 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:22:04.712177 16030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:04.712352 16015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:22:04.712440 16015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "6b543a3dbe1a4175ba59673d0823c960"
format_stamp: "Formatted at 2026-09-24 15:21:22 on dist-test-slave-zwbb"
server_key: "807003b2459e6e8fe525a8d962a4d694"
server_key_iv: "62d2f80a3ff272e713562caa880432a9"
server_key_version: "encryptionkey@0"
I20260924 15:22:04.712916 16015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:04.733377 16015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:04.735173 16015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:04.735278 16015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:04.739355 16015 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:38891
I20260924 15:22:04.739384 16083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:38891 every 8 connection(s)
I20260924 15:22:04.739739 16015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:04.739991 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 16015
I20260924 15:22:04.740368 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:42095
--local_ip_for_outbound_sockets=127.31.12.65
--tserver_master_addrs=127.31.12.126:38891
--webserver_port=34099
--webserver_interface=127.31.12.65
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:22:04.742029 16084 sys_catalog.cc:263] Verifying existing consensus state
I20260924 15:22:04.742682 16084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960: Bootstrap starting.
I20260924 15:22:04.754385 16084 log.cc:824] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:04.757071 16084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:22:04.757408 16084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960: Bootstrap complete.
I20260924 15:22:04.760605 16084 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b543a3dbe1a4175ba59673d0823c960" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38891 } }
I20260924 15:22:04.760998 16084 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b543a3dbe1a4175ba59673d0823c960, State: Initialized, Role: FOLLOWER
I20260924 15:22:04.761191 16084 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b543a3dbe1a4175ba59673d0823c960" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38891 } }
I20260924 15:22:04.761271 16084 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:04.761309 16084 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:04.761354 16084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 2 FOLLOWER]: Advancing to term 3
I20260924 15:22:04.762621 16084 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b543a3dbe1a4175ba59673d0823c960" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38891 } }
I20260924 15:22:04.762727 16084 leader_election.cc:302] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b543a3dbe1a4175ba59673d0823c960; no voters:
I20260924 15:22:04.762897 16084 leader_election.cc:288] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 15:22:04.763036 16088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 15:22:04.763161 16084 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:04.763217 16088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [term 3 LEADER]: Becoming Leader. State: Replica: 6b543a3dbe1a4175ba59673d0823c960, State: Running, Role: LEADER
I20260924 15:22:04.763331 16088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b543a3dbe1a4175ba59673d0823c960" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38891 } }
I20260924 15:22:04.763723 16089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6b543a3dbe1a4175ba59673d0823c960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b543a3dbe1a4175ba59673d0823c960" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38891 } } }
I20260924 15:22:04.763804 16089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:04.764035 16090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b543a3dbe1a4175ba59673d0823c960. Latest consensus state: current_term: 3 leader_uuid: "6b543a3dbe1a4175ba59673d0823c960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b543a3dbe1a4175ba59673d0823c960" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38891 } } }
I20260924 15:22:04.764173 16090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:04.764242 16098 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:04.765050 16098 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:04.765156 16098 catalog_manager.cc:1259] Loaded cluster ID: e475393132694bdca06419820774abaa
I20260924 15:22:04.765189 16098 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:04.766350 16098 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:04.766538 16098 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960: Loaded TSK: 0
I20260924 15:22:04.766588 16098 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6b543a3dbe1a4175ba59673d0823c960: Loaded TSK: 1
F20260924 15:22:04.766806 16098 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f40d76d09ad google::LogMessage::Fail() at ??:0
@ 0x7f40d76d388a google::LogMessage::SendToLog() at ??:0
@ 0x7f40d76d04de google::LogMessage::Flush() at ??:0
@ 0x7f40d76d1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f40d9e3d104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f40d9e3ad41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f40df62a1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f40df62a48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f40df62a594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f40df5f0c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f40df5faad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f40d959e0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f40d9593954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f40da3f76db start_thread at ??:0
@ 0x7f40d61ce61f clone at ??:0
W20260924 15:22:04.838761 16086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:04.838972 16086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:04.839011 16086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:04.840051 16086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:04.840116 16086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:04.840160 16086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:22:04.841632 16086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:42095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=34099
--tserver_master_addrs=127.31.12.126:38891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:04.841881 16086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:04.842095 16086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:04.846801 16126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:04.846797 16129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:04.846796 16127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:04.847115 16086 server_base.cc:1034] running on GCE node
I20260924 15:22:04.847307 16086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:04.847548 16086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:04.848714 16086 hybrid_clock.cc:648] HybridClock initialized: now 1790263324848701 us; error 27 us; skew 500 ppm
Sep 24 15:22:04 dist-test-slave-zwbb krb5kdc[14919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263324, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:04.851451 16086 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:22:04.851897 16086 webserver.cc:460] Webserver started at http://127.31.12.65:34099/ using document root <none> and password file <none>
I20260924 15:22:04.852138 16086 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:04.852183 16086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:04.853365 16086 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:22:04.854054 16136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:04.854234 16086 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:22:04.854321 16086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "c8795445a8444291aaf5234a6f1ca235"
format_stamp: "Formatted at 2026-09-24 15:21:23 on dist-test-slave-zwbb"
server_key: "dfd4ee048e2f19e023497f537bd3292a"
server_key_iv: "875d18effd49d0509d57fab07ce9bc37"
server_key_version: "encryptionkey@0"
I20260924 15:22:04.854671 16086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:04.890324 16086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:04.893064 16086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:04.893227 16086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:04.893496 16086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:04.893967 16143 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 15:22:04.895917 16086 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 15:22:04.895978 16086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 15:22:04.896021 16086 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 15:22:04.896636 16086 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 15:22:04.896689 16086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 15:22:04.896781 16143 tablet_bootstrap.cc:492] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235: Bootstrap starting.
I20260924 15:22:04.904060 16086 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:42095
I20260924 15:22:04.904433 16086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:22:04.905604 16256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:42095 every 8 connection(s)
I20260924 15:22:04.906906 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 16086
I20260924 15:22:04.907223 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:33753
--local_ip_for_outbound_sockets=127.31.12.66
--tserver_master_addrs=127.31.12.126:38891
--webserver_port=38315
--webserver_interface=127.31.12.66
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:22:04.908073 16143 log.cc:824] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:04.910746 16143 tablet_bootstrap.cc:492] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 15:22:04.911056 16143 tablet_bootstrap.cc:492] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235: Bootstrap complete.
I20260924 15:22:04.912302 16143 ts_tablet_manager.cc:1397] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.011s
I20260924 15:22:04.913655 16143 raft_consensus.cc:348] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } }
I20260924 15:22:04.914016 16143 raft_consensus.cc:729] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8795445a8444291aaf5234a6f1ca235, State: Initialized, Role: FOLLOWER
I20260924 15:22:04.914180 16143 consensus_queue.cc:260] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } }
I20260924 15:22:04.914428 16143 ts_tablet_manager.cc:1428] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260924 15:22:04.967594 16259 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:04.905085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.65:53027)
0924 15:22:04.905248 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:04.905290 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:04.905922 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:04.967416 (+ 61494us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
W20260924 15:22:04.967880 16257 heartbeater.cc:644] Failed to heartbeat to 127.31.12.126:38891 (0 consecutive failures): Network error: Failed to ping master at 127.31.12.126:38891: Client connection negotiation failed: client connection to 127.31.12.126:38891: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 15:22:04.968230 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:04.968122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.65:59231)
0924 15:22:04.968172 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:04.968212 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 15:22:04.968595 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:04.968518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.65:44577)
0924 15:22:04.968558 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:04.968580 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 15:22:04.968705 16257 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 15:22:04.991376 16261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:04.991590 16261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:04.991626 16261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:04.992743 16261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:04.992801 16261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:04.992844 16261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:22:04.994331 16261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:33753
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=38315
--tserver_master_addrs=127.31.12.126:38891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:04.994588 16261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:04.994798 16261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:22:04.999655 16261 server_base.cc:1034] running on GCE node
W20260924 15:22:04.999619 16268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:04.999606 16269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:04.999620 16271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:05.000077 16261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:05.000265 16261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:05.001472 16261 hybrid_clock.cc:648] HybridClock initialized: now 1790263325001426 us; error 38 us; skew 500 ppm
Sep 24 15:22:04 dist-test-slave-zwbb krb5kdc[14919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263324, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:05.003799 16261 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:22:05.004221 16261 webserver.cc:460] Webserver started at http://127.31.12.66:38315/ using document root <none> and password file <none>
I20260924 15:22:05.004452 16261 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:05.004518 16261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:05.005837 16261 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:22:05.006510 16279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:05.006716 16261 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:22:05.006819 16261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "0dde3faf177d4fcbb16905769398d7fd"
format_stamp: "Formatted at 2026-09-24 15:21:23 on dist-test-slave-zwbb"
server_key: "e9593341322ff8e1f86dd06834ebd980"
server_key_iv: "8dd3239116d4fbc18db82283e5b568c3"
server_key_version: "encryptionkey@0"
I20260924 15:22:05.007151 16261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:05.021252 16261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:05.023130 16261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:05.023278 16261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:05.023491 16261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:05.023803 16261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:22:05.023849 16261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:05.023893 16261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:22:05.023921 16261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:05.029631 16261 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:33753
I20260924 15:22:05.029717 16399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:33753 every 8 connection(s)
I20260924 15:22:05.030045 16261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:22:05.031020 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:05.030663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.66:58347)
0924 15:22:05.030814 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:05.030883 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
W20260924 15:22:05.031400 16400 heartbeater.cc:644] Failed to heartbeat to 127.31.12.126:38891 (0 consecutive failures): Network error: Failed to ping master at 127.31.12.126:38891: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
I20260924 15:22:05.031657 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:05.031567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.66:60433)
0924 15:22:05.031610 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:05.031635 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 15:22:05.031942 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:05.031849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.66:42569)
0924 15:22:05.031901 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:05.031923 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260924 15:22:05.032044 16400 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 15:22:05.034031 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 16261
I20260924 15:22:05.034274 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:33089
--local_ip_for_outbound_sockets=127.31.12.67
--tserver_master_addrs=127.31.12.126:38891
--webserver_port=46271
--webserver_interface=127.31.12.67
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:34979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:05.114358 16403 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:05.114598 16403 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:05.114637 16403 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:05.115778 16403 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:05.115839 16403 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:05.115885 16403 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:22:05.117304 16403 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:34979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:33089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=46271
--tserver_master_addrs=127.31.12.126:38891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16403
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:05.117574 16403 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:05.117791 16403 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:05.122699 16411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:05.122759 16409 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:05.122711 16408 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:05.123057 16403 server_base.cc:1034] running on GCE node
I20260924 15:22:05.123255 16403 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:05.123492 16403 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:05.124645 16403 hybrid_clock.cc:648] HybridClock initialized: now 1790263325124636 us; error 33 us; skew 500 ppm
Sep 24 15:22:05 dist-test-slave-zwbb krb5kdc[14919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263325, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:05.127115 16403 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:22:05.127595 16403 webserver.cc:460] Webserver started at http://127.31.12.67:46271/ using document root <none> and password file <none>
I20260924 15:22:05.127811 16403 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:05.127875 16403 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:05.129201 16403 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:22:05.129819 16418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:05.129976 16403 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:05.130070 16403 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "0c339766ca354ae88b148dc07f93b0da"
format_stamp: "Formatted at 2026-09-24 15:21:23 on dist-test-slave-zwbb"
server_key: "1eabcafc97b76446cf215ad1cdd19a3f"
server_key_iv: "396a3591f02a627a588eb0b1a2729b76"
server_key_version: "encryptionkey@0"
I20260924 15:22:05.130419 16403 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:05.143924 16403 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:05.146291 16403 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:05.146452 16403 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:05.146680 16403 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:05.146982 16403 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:22:05.147033 16403 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:05.147096 16403 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:22:05.147137 16403 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:05.153069 16403 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:33089
I20260924 15:22:05.153121 16532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:33089 every 8 connection(s)
I20260924 15:22:05.153458 16403 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:22:05.154487 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:05.154132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.67:49899)
0924 15:22:05.154279 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:05.154348 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
W20260924 15:22:05.154923 16533 heartbeater.cc:644] Failed to heartbeat to 127.31.12.126:38891 (0 consecutive failures): Network error: Failed to ping master at 127.31.12.126:38891: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
I20260924 15:22:05.155220 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:05.155135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.67:46695)
0924 15:22:05.155179 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:05.155204 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 15:22:05.155483 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:05.155434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.67:46831)
0924 15:22:05.155455 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:05.155472 (+ 17us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260924 15:22:05.155583 16533 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 15:22:05.159073 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 16403
I20260924 15:22:05.969252 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:05.969024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.65:34723)
0924 15:22:05.969151 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:05.969186 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":61,"threads_started":1}
I20260924 15:22:06.032575 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:06.032334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.66:36567)
0924 15:22:06.032476 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:06.032509 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":58,"threads_started":1}
I20260924 15:22:06.120121 16539 raft_consensus.cc:482] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:22:06.120271 16539 raft_consensus.cc:504] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } }
I20260924 15:22:06.120546 16539 leader_election.cc:288] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c339766ca354ae88b148dc07f93b0da (127.31.12.67:33089), 0dde3faf177d4fcbb16905769398d7fd (127.31.12.66:33753)
Sep 24 15:22:06 dist-test-slave-zwbb krb5kdc[14919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263324, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.66@KRBTEST.COM
Sep 24 15:22:06 dist-test-slave-zwbb krb5kdc[14919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263324, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.67@KRBTEST.COM
I20260924 15:22:06.136960 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:06.120837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:33089 (local address 127.31.12.65:59199)
0924 15:22:06.120916 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:06.120931 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:06.121005 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:06.122105 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:06.122108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:06.122705 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:06.122995 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:06.123003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:06.124687 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:06.124688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:06.126155 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:06.126156 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:06.126968 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:06.126969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:06.127115 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:06.131312 (+ 4197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:06.131323 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:06.133577 (+ 2254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:06.135449 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:06.135450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:06.135450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:06.135681 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:06.135951 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:06.135951 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:06.135952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:06.136057 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:06.136452 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:06.136454 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:06.136561 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 15:22:06.136663 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 15:22:06.136860 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":80}
I20260924 15:22:06.136974 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:06.121123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:44693 (local address 127.31.12.66:33753)
0924 15:22:06.121275 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:06.121282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:06.121309 (+ 27us) server_negotiation.cc:407] Connection header received
0924 15:22:06.121853 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:06.121855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:06.121893 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:06.122022 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:06.123121 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:06.124539 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:06.126248 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:06.126861 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:06.126975 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:06.133312 (+ 6337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:06.133316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:06.133317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:06.133330 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:06.135268 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:06.135770 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:06.135773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:06.135774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:06.135866 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:06.136122 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:06.136123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:06.136123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:06.136354 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:06.136436 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:06.136716 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 15:22:06.136848 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260924 15:22:06.137024 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:06.120952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:59199 (local address 127.31.12.67:33089)
0924 15:22:06.121128 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:06.121138 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:06.121160 (+ 22us) server_negotiation.cc:407] Connection header received
0924 15:22:06.121780 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:06.121784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:06.121864 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:06.122023 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:06.123121 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:06.124539 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:06.126244 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:06.126861 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:06.126989 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:06.133673 (+ 6684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:06.133677 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:06.133677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:06.133693 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:06.135345 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:06.135779 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:06.135782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:06.135783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:06.135868 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:06.136132 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:06.136133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:06.136134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:06.136355 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:06.136456 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:06.136720 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 15:22:06.136910 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260924 15:22:06.137070 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:06.120975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:33753 (local address 127.31.12.65:44693)
0924 15:22:06.121122 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:06.121133 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:06.121178 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:06.122104 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:06.122108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:06.122705 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:06.122995 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:06.123003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:06.124674 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:06.124675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:06.126153 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:06.126156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:06.126967 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:06.126968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:06.127115 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:06.131312 (+ 4197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:06.131323 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:06.133204 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:06.135351 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:06.135359 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:06.135361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:06.135683 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:06.135931 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:06.135932 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:06.135933 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:06.136057 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:06.136423 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:06.136430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:06.136562 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 15:22:06.136665 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 15:22:06.136943 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":202,"thread_start_us":77,"threads_started":1}
W20260924 15:22:06.138516 16138 leader_election.cc:341] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0dde3faf177d4fcbb16905769398d7fd (127.31.12.66:33753): Not found: Tablet not found: e789a8c0c13e49dcbf114c96d28096dc
W20260924 15:22:06.138612 16138 leader_election.cc:341] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c339766ca354ae88b148dc07f93b0da (127.31.12.67:33089): Not found: Tablet not found: e789a8c0c13e49dcbf114c96d28096dc
I20260924 15:22:06.138652 16138 leader_election.cc:302] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8795445a8444291aaf5234a6f1ca235; no voters: 0c339766ca354ae88b148dc07f93b0da, 0dde3faf177d4fcbb16905769398d7fd
I20260924 15:22:06.138773 16539 raft_consensus.cc:2726] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:22:06.156066 16543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:06.155877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.67:41231)
0924 15:22:06.155995 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:06.156035 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260924 15:22:06.969894 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:06.969684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.65:58143)
0924 15:22:06.969807 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:06.969846 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260924 15:22:07.033190 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:07.033014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.66:37323)
0924 15:22:07.033119 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:07.033153 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":53,"threads_started":1}
I20260924 15:22:07.156723 16546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:07.156531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.67:55315)
0924 15:22:07.156649 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:07.156685 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260924 15:22:07.903205 16547 raft_consensus.cc:482] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:22:07.903306 16547 raft_consensus.cc:504] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } }
I20260924 15:22:07.903501 16547 leader_election.cc:288] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c339766ca354ae88b148dc07f93b0da (127.31.12.67:33089), 0dde3faf177d4fcbb16905769398d7fd (127.31.12.66:33753)
W20260924 15:22:07.903900 16138 leader_election.cc:341] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0dde3faf177d4fcbb16905769398d7fd (127.31.12.66:33753): Not found: Tablet not found: e789a8c0c13e49dcbf114c96d28096dc
W20260924 15:22:07.903967 16138 leader_election.cc:341] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c339766ca354ae88b148dc07f93b0da (127.31.12.67:33089): Not found: Tablet not found: e789a8c0c13e49dcbf114c96d28096dc
I20260924 15:22:07.903985 16138 leader_election.cc:302] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8795445a8444291aaf5234a6f1ca235; no voters: 0c339766ca354ae88b148dc07f93b0da, 0dde3faf177d4fcbb16905769398d7fd
I20260924 15:22:07.904070 16547 raft_consensus.cc:2726] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:22:07.970556 16548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:07.970344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.65:52549)
0924 15:22:07.970468 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:07.970504 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260924 15:22:08.033897 16549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:08.033689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.66:58455)
0924 15:22:08.033837 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:08.033865 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
I20260924 15:22:08.157344 16550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:08.157151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.67:50201)
0924 15:22:08.157269 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:08.157308 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260924 15:22:08.971289 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:08.971045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.65:41391)
0924 15:22:08.971195 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:08.971245 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260924 15:22:09.034576 16552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:09.034317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.66:59741)
0924 15:22:09.034494 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:09.034529 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":67,"threads_started":1}
I20260924 15:22:09.157974 16553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:09.157766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.67:40413)
0924 15:22:09.157896 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:09.157931 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":95,"threads_started":1}
I20260924 15:22:09.971973 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:09.971742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.65:51631)
0924 15:22:09.971868 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:09.971918 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
I20260924 15:22:10.035168 16555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:10.034972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.66:51049)
0924 15:22:10.035099 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:10.035130 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":59,"threads_started":1}
I20260924 15:22:10.158644 16556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:10.158437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.67:45419)
0924 15:22:10.158564 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:10.158603 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260924 15:22:10.746372 16557 raft_consensus.cc:482] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:22:10.746475 16557 raft_consensus.cc:504] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } }
I20260924 15:22:10.746688 16557 leader_election.cc:288] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c339766ca354ae88b148dc07f93b0da (127.31.12.67:33089), 0dde3faf177d4fcbb16905769398d7fd (127.31.12.66:33753)
W20260924 15:22:10.747123 16138 leader_election.cc:341] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c339766ca354ae88b148dc07f93b0da (127.31.12.67:33089): Not found: Tablet not found: e789a8c0c13e49dcbf114c96d28096dc
W20260924 15:22:10.747201 16138 leader_election.cc:341] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0dde3faf177d4fcbb16905769398d7fd (127.31.12.66:33753): Not found: Tablet not found: e789a8c0c13e49dcbf114c96d28096dc
I20260924 15:22:10.747244 16138 leader_election.cc:302] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8795445a8444291aaf5234a6f1ca235; no voters: 0c339766ca354ae88b148dc07f93b0da, 0dde3faf177d4fcbb16905769398d7fd
I20260924 15:22:10.747354 16557 raft_consensus.cc:2726] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:22:10.972759 16558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:10.972513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.65:45147)
0924 15:22:10.972674 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:10.972713 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":96,"threads_started":1}
I20260924 15:22:11.035948 16559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:11.035706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.66:58923)
0924 15:22:11.035863 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:11.035906 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260924 15:22:11.159368 16560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:11.159139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.67:52199)
0924 15:22:11.159287 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:11.159324 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260924 15:22:11.973534 16561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:11.973274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.65:43859)
0924 15:22:11.973402 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:11.973482 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260924 15:22:12.036613 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:12.036391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.66:39579)
0924 15:22:12.036533 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:12.036573 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
I20260924 15:22:12.160015 16563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:12.159790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.67:56949)
0924 15:22:12.159935 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:12.159972 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260924 15:22:12.974180 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:12.973971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.65:52759)
0924 15:22:12.974099 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:12.974134 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260924 15:22:13.037279 16565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:13.037083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.66:34291)
0924 15:22:13.037206 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:13.037242 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":57,"threads_started":1}
I20260924 15:22:13.160728 16566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:13.160494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.67:34093)
0924 15:22:13.160650 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:13.160687 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
I20260924 15:22:13.975004 16567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:13.974742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.65:51229)
0924 15:22:13.974902 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:13.974955 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260924 15:22:14.038014 16568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:14.037771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.66:52725)
0924 15:22:14.037928 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:14.037972 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":74,"threads_started":1}
I20260924 15:22:14.067970 16569 raft_consensus.cc:482] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:22:14.068100 16569 raft_consensus.cc:504] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c339766ca354ae88b148dc07f93b0da" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33089 } } peers { permanent_uuid: "0dde3faf177d4fcbb16905769398d7fd" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33753 } } peers { permanent_uuid: "c8795445a8444291aaf5234a6f1ca235" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42095 } }
I20260924 15:22:14.068327 16569 leader_election.cc:288] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c339766ca354ae88b148dc07f93b0da (127.31.12.67:33089), 0dde3faf177d4fcbb16905769398d7fd (127.31.12.66:33753)
W20260924 15:22:14.068791 16138 leader_election.cc:341] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c339766ca354ae88b148dc07f93b0da (127.31.12.67:33089): Not found: Tablet not found: e789a8c0c13e49dcbf114c96d28096dc
W20260924 15:22:14.068892 16138 leader_election.cc:341] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0dde3faf177d4fcbb16905769398d7fd (127.31.12.66:33753): Not found: Tablet not found: e789a8c0c13e49dcbf114c96d28096dc
I20260924 15:22:14.068920 16138 leader_election.cc:302] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c8795445a8444291aaf5234a6f1ca235; no voters: 0c339766ca354ae88b148dc07f93b0da, 0dde3faf177d4fcbb16905769398d7fd
I20260924 15:22:14.069069 16569 raft_consensus.cc:2726] T e789a8c0c13e49dcbf114c96d28096dc P c8795445a8444291aaf5234a6f1ca235 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 15:22:14.161615 16570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:14.161314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.67:56621)
0924 15:22:14.161474 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:14.161559 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":76,"threads_started":1}
I20260924 15:22:14.975826 16571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:14.975566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.65:35989)
0924 15:22:14.975729 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:14.975777 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":61,"threads_started":1}
I20260924 15:22:15.038741 16572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:15.038510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.66:41243)
0924 15:22:15.038657 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:15.038702 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260924 15:22:15.162364 16573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:15.162100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.67:34099)
0924 15:22:15.162260 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:15.162319 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":61,"threads_started":1}
I20260924 15:22:15.976570 16574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:15.976351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.65:42593)
0924 15:22:15.976493 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:15.976526 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
I20260924 15:22:16.039592 16575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:16.039322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.66:38003)
0924 15:22:16.039511 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:16.039549 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":89,"threads_started":1}
I20260924 15:22:16.163033 16577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:16.162831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.67:40587)
0924 15:22:16.162956 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:16.162990 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260924 15:22:16.977347 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:16.977091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.65:48129)
0924 15:22:16.977253 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:16.977300 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":66,"threads_started":1}
I20260924 15:22:17.040323 16579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:17.040096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.66:51933)
0924 15:22:17.040235 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:17.040277 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260924 15:22:17.163693 16580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:17.163453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.67:37291)
0924 15:22:17.163605 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:17.163648 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":56,"threads_started":1}
I20260924 15:22:17.978091 16581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:17.977850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.65:50361)
0924 15:22:17.977996 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:17.978045 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260924 15:22:18.041020 16582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:18.040804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.66:45055)
0924 15:22:18.040926 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:18.040961 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260924 15:22:18.164373 16583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:18.164174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.67:50743)
0924 15:22:18.164290 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:18.164326 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260924 15:22:18.978754 16584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:18.978559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.65:58657)
0924 15:22:18.978676 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:18.978710 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260924 15:22:19.041750 16585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:19.041557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.66:57545)
0924 15:22:19.041673 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:19.041709 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":54,"threads_started":1}
I20260924 15:22:19.165084 16586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:19.164873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.67:34997)
0924 15:22:19.165001 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:19.165037 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260924 15:22:19.979457 16587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:19.979238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.65:50825)
0924 15:22:19.979377 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:19.979412 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260924 15:22:20.042521 16588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:20.042286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.66:53609)
0924 15:22:20.042444 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:20.042477 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260924 15:22:20.164995 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 16086
I20260924 15:22:20.166831 16589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:20.166023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38891 (local address 127.31.12.67:48071)
0924 15:22:20.166729 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:20.166780 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.126:38891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":49,"threads_started":1}
I20260924 15:22:20.170032 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 16261
I20260924 15:22:20.174664 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 16403
2026-09-24T15:22:20Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58593 ms)
[----------] 30 tests from SecurityITest (259875 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:20 dist-test-slave-zwbb krb5kdc[16592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:20 dist-test-slave-zwbb krb5kdc[16592](info): set up 2 sockets
Sep 24 15:22:20 dist-test-slave-zwbb krb5kdc[16592](info): commencing operation
krb5kdc: starting...
W20260924 15:22:21.282878 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:21 dist-test-slave-zwbb krb5kdc[16592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:22:21.400007 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:46023
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:46023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:21.478226 16606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:21.478444 16606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:21.478487 16606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:21.479668 16606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:21.479734 16606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:21.479758 16606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:21.479799 16606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:21.479827 16606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:21.481249 16606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:46023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:46023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:21.481556 16606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:21.481786 16606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:21.486845 16614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:21.486817 16612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:21.486801 16611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:21.486887 16606 server_base.cc:1034] running on GCE node
I20260924 15:22:21.487298 16606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:21.487591 16606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:21.488734 16606 hybrid_clock.cc:648] HybridClock initialized: now 1790263341488709 us; error 46 us; skew 500 ppm
Sep 24 15:22:21 dist-test-slave-zwbb krb5kdc[16592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263341, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:21.491214 16606 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:22:21.491588 16606 webserver.cc:460] Webserver started at http://127.31.12.126:35469/ using document root <none> and password file <none>
I20260924 15:22:21.491752 16606 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:21.491794 16606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:21.491917 16606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:21.492926 16606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "c86e1a817d1846ae8e25bcd03d04c5cc"
format_stamp: "Formatted at 2026-09-24 15:22:21 on dist-test-slave-zwbb"
server_key: "23bb01e76aba9536e71627998ed383ca"
server_key_iv: "824ccc243766812e574639a5ae486079"
server_key_version: "encryptionkey@0"
I20260924 15:22:21.493222 16606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "c86e1a817d1846ae8e25bcd03d04c5cc"
format_stamp: "Formatted at 2026-09-24 15:22:21 on dist-test-slave-zwbb"
server_key: "23bb01e76aba9536e71627998ed383ca"
server_key_iv: "824ccc243766812e574639a5ae486079"
server_key_version: "encryptionkey@0"
I20260924 15:22:21.494650 16606 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:22:21.495390 16621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:21.495566 16606 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:21.495648 16606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "c86e1a817d1846ae8e25bcd03d04c5cc"
format_stamp: "Formatted at 2026-09-24 15:22:21 on dist-test-slave-zwbb"
server_key: "23bb01e76aba9536e71627998ed383ca"
server_key_iv: "824ccc243766812e574639a5ae486079"
server_key_version: "encryptionkey@0"
I20260924 15:22:21.495702 16606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:21.504917 16606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:21.506779 16606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:21.506880 16606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:21.510501 16606 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:46023
I20260924 15:22:21.510578 16673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:46023 every 8 connection(s)
I20260924 15:22:21.510927 16606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:21.511737 16674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:21.513876 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 16606
I20260924 15:22:21.513967 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:21.513967 16674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc: Bootstrap starting.
I20260924 15:22:21.514155 31793 external_mini_cluster.cc:1428] Setting key 09912bcd4090bf1ccd3c0db3a4f9a9e0
I20260924 15:22:21.514665 16674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:21.514993 16674 log.cc:824] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:21.515699 16674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc: No bootstrap required, opened a new log
I20260924 15:22:21.517725 16674 raft_consensus.cc:348] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86e1a817d1846ae8e25bcd03d04c5cc" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 46023 } }
I20260924 15:22:21.517834 16674 raft_consensus.cc:374] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:21.517853 16674 raft_consensus.cc:729] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86e1a817d1846ae8e25bcd03d04c5cc, State: Initialized, Role: FOLLOWER
I20260924 15:22:21.517977 16674 consensus_queue.cc:260] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86e1a817d1846ae8e25bcd03d04c5cc" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 46023 } }
I20260924 15:22:21.518043 16674 raft_consensus.cc:388] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:21.518064 16674 raft_consensus.cc:482] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:21.518105 16674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:21.518685 16674 raft_consensus.cc:504] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86e1a817d1846ae8e25bcd03d04c5cc" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 46023 } }
I20260924 15:22:21.518781 16674 leader_election.cc:302] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c86e1a817d1846ae8e25bcd03d04c5cc; no voters:
I20260924 15:22:21.518930 16674 leader_election.cc:288] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:21.518976 16679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:21.519124 16679 raft_consensus.cc:686] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc [term 1 LEADER]: Becoming Leader. State: Replica: c86e1a817d1846ae8e25bcd03d04c5cc, State: Running, Role: LEADER
I20260924 15:22:21.519220 16674 sys_catalog.cc:564] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:21.519245 16679 consensus_queue.cc:237] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86e1a817d1846ae8e25bcd03d04c5cc" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 46023 } }
I20260924 15:22:21.519583 16679 sys_catalog.cc:455] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c86e1a817d1846ae8e25bcd03d04c5cc. Latest consensus state: current_term: 1 leader_uuid: "c86e1a817d1846ae8e25bcd03d04c5cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86e1a817d1846ae8e25bcd03d04c5cc" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 46023 } } }
I20260924 15:22:21.519598 16680 sys_catalog.cc:455] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c86e1a817d1846ae8e25bcd03d04c5cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c86e1a817d1846ae8e25bcd03d04c5cc" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 46023 } } }
I20260924 15:22:21.519693 16679 sys_catalog.cc:458] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:21.519701 16680 sys_catalog.cc:458] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:21.520535 16688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 15:22:21 dist-test-slave-zwbb krb5kdc[16592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:22:21.521145 16688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:21.522722 16688 catalog_manager.cc:1347] Generated new cluster ID: 22b97ff4ec234a998d51fb42f681a649
I20260924 15:22:21.522778 16688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:21.524713 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:21.514687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53032 (local address 127.31.12.126:46023)
0924 15:22:21.514872 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:21.514880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:21.514910 (+ 30us) server_negotiation.cc:407] Connection header received
0924 15:22:21.515446 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:21.515454 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:21.515531 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:21.515664 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:21.516236 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:21.517477 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:21.518280 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:21.518854 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:21.518959 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:21.521531 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:21.521535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:21.521535 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:21.521546 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:21.523258 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:21.523632 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:21.523635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:21.523635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:21.523726 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:21.523945 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:21.523946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:21.523946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:21.524161 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:21.524314 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:21.524383 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 15:22:21.524519 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":74,"threads_started":1}
I20260924 15:22:21.543458 16688 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:21.543982 16688 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:21.550618 16688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc: Generated new TSK 0
I20260924 15:22:21.550793 16688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
I20260924 15:22:21.584230 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:46023
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:21.678428 16700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:21.678663 16700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:21.678689 16700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:21.679801 16700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:21.679859 16700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:21.679913 16700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:22:21.681370 16700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:46023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:21.681596 16700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:21.681780 16700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:21.686691 16705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:21.686698 16706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:21.686751 16700 server_base.cc:1034] running on GCE node
W20260924 15:22:21.686712 16708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:21.687103 16700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:21.687299 16700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:21.688434 16700 hybrid_clock.cc:648] HybridClock initialized: now 1790263341688414 us; error 37 us; skew 500 ppm
Sep 24 15:22:21 dist-test-slave-zwbb krb5kdc[16592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263341, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:21.690826 16700 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:22:21.691215 16700 webserver.cc:460] Webserver started at http://127.31.12.65:36107/ using document root <none> and password file <none>
I20260924 15:22:21.691431 16700 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:21.691494 16700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:21.691632 16700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:21.692574 16700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "bf877393a4094e9c999821c0dfa66e2e"
format_stamp: "Formatted at 2026-09-24 15:22:21 on dist-test-slave-zwbb"
server_key: "e97dcfc21279234dce142acb7d12f8d4"
server_key_iv: "60b8bca1855ddcaa5b8e3d398d497cac"
server_key_version: "encryptionkey@0"
I20260924 15:22:21.692952 16700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "bf877393a4094e9c999821c0dfa66e2e"
format_stamp: "Formatted at 2026-09-24 15:22:21 on dist-test-slave-zwbb"
server_key: "e97dcfc21279234dce142acb7d12f8d4"
server_key_iv: "60b8bca1855ddcaa5b8e3d398d497cac"
server_key_version: "encryptionkey@0"
I20260924 15:22:21.694317 16700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:22:21.695065 16715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:21.695238 16700 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:21.695327 16700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "bf877393a4094e9c999821c0dfa66e2e"
format_stamp: "Formatted at 2026-09-24 15:22:21 on dist-test-slave-zwbb"
server_key: "e97dcfc21279234dce142acb7d12f8d4"
server_key_iv: "60b8bca1855ddcaa5b8e3d398d497cac"
server_key_version: "encryptionkey@0"
I20260924 15:22:21.695407 16700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:21.717196 16700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:21.719300 16700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:21.719455 16700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:21.719681 16700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:21.719983 16700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:22:21.720037 16700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:21.720077 16700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:22:21.720103 16700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:21.725914 16700 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:45091
I20260924 15:22:21.726029 16829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:45091 every 8 connection(s)
I20260924 15:22:21.726308 16700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:22:21.729298 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 16700
I20260924 15:22:21.729421 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:22:21.729624 31793 external_mini_cluster.cc:1428] Setting key c357e5e838530967e43e00e15738d2fe
Sep 24 15:22:21 dist-test-slave-zwbb krb5kdc[16592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263341, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:22:21.736279 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:21.727030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:39945 (local address 127.31.12.126:46023)
0924 15:22:21.727097 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:21.727098 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:21.727314 (+ 216us) server_negotiation.cc:407] Connection header received
0924 15:22:21.728097 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:21.728098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:21.728130 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:21.728185 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:21.729182 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:21.729817 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:21.730826 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:21.731049 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:21.731125 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:21.733421 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:21.733422 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:21.733422 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:21.733430 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:21.734831 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:21.735267 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:21.735267 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:21.735268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:21.735331 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:21.735576 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:21.735577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:21.735577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:21.735700 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:21.735773 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:21.736107 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 15:22:21.736187 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 15:22:21.736488 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:21.727027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:46023 (local address 127.31.12.65:39945)
0924 15:22:21.727200 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:21.727235 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:21.727984 (+ 749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:21.728252 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:21.728256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:21.728878 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:21.729064 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:21.729069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:21.729903 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:21.729905 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:21.730726 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:21.730728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:21.731133 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:21.731134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:21.731271 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:21.731821 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:21.731828 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:21.733316 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:21.734938 (+ 1622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:21.734946 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:21.734948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:21.735170 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:21.735398 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:21.735399 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:21.735400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:21.735504 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:21.735764 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:21.735771 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:21.735960 (+ 189us) client_negotiation.cc:769] Sending connection context
0924 15:22:21.736085 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 15:22:21.736242 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260924 15:22:21.737428 16830 heartbeater.cc:347] Connected to a master server at 127.31.12.126:46023
I20260924 15:22:21.737560 16830 heartbeater.cc:464] Registering TS with master...
I20260924 15:22:21.737728 16830 heartbeater.cc:511] Master 127.31.12.126:46023 requested a full tablet report, sending...
I20260924 15:22:21.738318 16638 ts_manager.cc:194] Registered new tserver with Master: bf877393a4094e9c999821c0dfa66e2e (127.31.12.65:45091)
WARNING: no policy specified for kudu/127.31.12.66@KRBTEST.COM; defaulting to no policy
I20260924 15:22:21.739077 16638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:39945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
I20260924 15:22:21.754482 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:46023
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:21.832954 16835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:21.833133 16835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:21.833156 16835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:21.834295 16835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:21.834334 16835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:21.834386 16835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:22:21.835721 16835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:46023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:21.835984 16835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:21.836210 16835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:21.841070 16841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:21.841167 16835 server_base.cc:1034] running on GCE node
W20260924 15:22:21.841042 16840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:21.841060 16843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:21.841430 16835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:21.841719 16835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:21.842865 16835 hybrid_clock.cc:648] HybridClock initialized: now 1790263341842852 us; error 28 us; skew 500 ppm
Sep 24 15:22:21 dist-test-slave-zwbb krb5kdc[16592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263341, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:21.845274 16835 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:22:21.845731 16835 webserver.cc:460] Webserver started at http://127.31.12.66:42715/ using document root <none> and password file <none>
I20260924 15:22:21.845958 16835 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:21.846002 16835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:21.846146 16835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:21.847131 16835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "3bf15ee9071a405cb9677d9fb6aabe3a"
format_stamp: "Formatted at 2026-09-24 15:22:21 on dist-test-slave-zwbb"
server_key: "28c956b20c4c3139106ef774c4c90411"
server_key_iv: "fce0cf298be46501f462a627a74c82f0"
server_key_version: "encryptionkey@0"
I20260924 15:22:21.847469 16835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "3bf15ee9071a405cb9677d9fb6aabe3a"
format_stamp: "Formatted at 2026-09-24 15:22:21 on dist-test-slave-zwbb"
server_key: "28c956b20c4c3139106ef774c4c90411"
server_key_iv: "fce0cf298be46501f462a627a74c82f0"
server_key_version: "encryptionkey@0"
I20260924 15:22:21.848724 16835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:22:21.849578 16850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:21.849773 16835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:22:21.849859 16835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "3bf15ee9071a405cb9677d9fb6aabe3a"
format_stamp: "Formatted at 2026-09-24 15:22:21 on dist-test-slave-zwbb"
server_key: "28c956b20c4c3139106ef774c4c90411"
server_key_iv: "fce0cf298be46501f462a627a74c82f0"
server_key_version: "encryptionkey@0"
I20260924 15:22:21.849936 16835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:21.861385 16835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:21.863060 16835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:21.863197 16835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:21.863420 16835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:21.863709 16835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:22:21.863744 16835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:21.863765 16835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:22:21.863776 16835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:21.869227 16835 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:33129
I20260924 15:22:21.869272 16967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:33129 every 8 connection(s)
I20260924 15:22:21.869618 16835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:22:21 dist-test-slave-zwbb krb5kdc[16592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263341, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:22:21.877826 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:21.870280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:50135 (local address 127.31.12.126:46023)
0924 15:22:21.870345 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:21.870348 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:21.870529 (+ 181us) server_negotiation.cc:407] Connection header received
0924 15:22:21.870967 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:21.870968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:21.870977 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:21.871027 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:21.871905 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:21.872383 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:21.873301 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:21.873568 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:21.873621 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:21.875548 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:21.875549 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:21.875549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:21.875556 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:21.876710 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:21.877031 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:21.877032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:21.877032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:21.877080 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:21.877264 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:21.877265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:21.877265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:21.877356 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:21.877427 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:21.877703 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 15:22:21.877765 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:22:21.877967 16970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:21.870251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:46023 (local address 127.31.12.66:50135)
0924 15:22:21.870416 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:21.870447 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:21.870889 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:21.871091 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:21.871094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:21.871649 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:21.871816 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:21.871820 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:21.872454 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:21.872455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:21.873223 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:21.873225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:21.873635 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:21.873635 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:21.873733 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:21.874222 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:21.874229 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:21.875468 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:21.876775 (+ 1307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:21.876781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:21.876782 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:21.876971 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:21.877134 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:21.877134 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:21.877135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:21.877216 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:21.877407 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:21.877411 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:21.877588 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 15:22:21.877668 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 15:22:21.877776 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260924 15:22:21.878443 16968 heartbeater.cc:347] Connected to a master server at 127.31.12.126:46023
I20260924 15:22:21.878541 16968 heartbeater.cc:464] Registering TS with master...
I20260924 15:22:21.878726 16968 heartbeater.cc:511] Master 127.31.12.126:46023 requested a full tablet report, sending...
I20260924 15:22:21.879149 16638 ts_manager.cc:194] Registered new tserver with Master: 3bf15ee9071a405cb9677d9fb6aabe3a (127.31.12.66:33129)
I20260924 15:22:21.879341 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 16835
I20260924 15:22:21.879463 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:22:21.879626 16638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:50135
I20260924 15:22:21.879673 31793 external_mini_cluster.cc:1428] Setting key 02e37c9826661b133a44dd5eeee32e3b
WARNING: no policy specified for kudu/127.31.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
I20260924 15:22:21.904134 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:46023
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:21.982414 16973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:21.982597 16973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:21.982620 16973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:21.983664 16973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:21.983697 16973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:21.983722 16973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:22:21.985064 16973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:46023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:21.985227 16973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:21.985383 16973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:21.990248 16978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:21.990271 16981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:21.990284 16979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:21.990409 16973 server_base.cc:1034] running on GCE node
I20260924 15:22:21.990691 16973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:21.990890 16973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:21.992030 16973 hybrid_clock.cc:648] HybridClock initialized: now 1790263341992019 us; error 29 us; skew 500 ppm
Sep 24 15:22:21 dist-test-slave-zwbb krb5kdc[16592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263341, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:21.994477 16973 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:22:21.994843 16973 webserver.cc:460] Webserver started at http://127.31.12.67:38859/ using document root <none> and password file <none>
I20260924 15:22:21.995030 16973 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:21.995070 16973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:21.995153 16973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:21.996141 16973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "0187f679583c4cf7a66dcef4e90fe004"
format_stamp: "Formatted at 2026-09-24 15:22:21 on dist-test-slave-zwbb"
server_key: "5a336f5f5ac775d864e8e70ac62d267d"
server_key_iv: "03e72b696e13ff65b65777b2ce6f42e1"
server_key_version: "encryptionkey@0"
I20260924 15:22:21.996436 16973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "0187f679583c4cf7a66dcef4e90fe004"
format_stamp: "Formatted at 2026-09-24 15:22:21 on dist-test-slave-zwbb"
server_key: "5a336f5f5ac775d864e8e70ac62d267d"
server_key_iv: "03e72b696e13ff65b65777b2ce6f42e1"
server_key_version: "encryptionkey@0"
I20260924 15:22:21.997804 16973 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:22:21.998605 16988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:21.998828 16973 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:22:21.998929 16973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "0187f679583c4cf7a66dcef4e90fe004"
format_stamp: "Formatted at 2026-09-24 15:22:21 on dist-test-slave-zwbb"
server_key: "5a336f5f5ac775d864e8e70ac62d267d"
server_key_iv: "03e72b696e13ff65b65777b2ce6f42e1"
server_key_version: "encryptionkey@0"
I20260924 15:22:21.999042 16973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:22.014268 16973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:22.015833 16973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:22.015928 16973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:22.016078 16973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:22.016361 16973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:22:22.016422 16973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:22.016472 16973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:22:22.016516 16973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:22.022630 16973 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:32887
I20260924 15:22:22.022667 17103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:32887 every 8 connection(s)
I20260924 15:22:22.022998 16973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:22:22.028716 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 16973
I20260924 15:22:22.028808 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:22:22.028990 31793 external_mini_cluster.cc:1428] Setting key 7019457570ed5ff24ec2cd20ec070c57
Sep 24 15:22:22 dist-test-slave-zwbb krb5kdc[16592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263341, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:22:22.032591 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.023700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:41067 (local address 127.31.12.126:46023)
0924 15:22:22.023763 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:22.023764 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:22.023977 (+ 213us) server_negotiation.cc:407] Connection header received
0924 15:22:22.024458 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:22.024459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:22.024472 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:22.024543 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:22.025738 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.026243 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.027187 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.027406 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.027488 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.029800 (+ 2312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:22.029801 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:22.029801 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:22.029809 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:22.031279 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:22.031624 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:22.031624 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:22.031625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:22.031672 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:22.031940 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:22.031942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:22.031942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:22.032085 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:22.032163 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:22.032426 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 15:22:22.032505 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 15:22:22.032796 17106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.023675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:46023 (local address 127.31.12.67:41067)
0924 15:22:22.023847 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:22.023880 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:22.024357 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:22.024611 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:22.024614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:22.025394 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:22.025635 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.025640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.026332 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.026333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.027095 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.027097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.027498 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.027499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.027611 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.028074 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:22.028082 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:22.029717 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:22.031345 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:22.031351 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:22.031352 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:22.031561 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:22.031728 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:22.031729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:22.031729 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:22.031839 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:22.032165 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:22.032172 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:22.032292 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 15:22:22.032390 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 15:22:22.032554 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260924 15:22:22.033213 17104 heartbeater.cc:347] Connected to a master server at 127.31.12.126:46023
I20260924 15:22:22.033290 17104 heartbeater.cc:464] Registering TS with master...
I20260924 15:22:22.033504 17104 heartbeater.cc:511] Master 127.31.12.126:46023 requested a full tablet report, sending...
I20260924 15:22:22.033918 16638 ts_manager.cc:194] Registered new tserver with Master: 0187f679583c4cf7a66dcef4e90fe004 (127.31.12.67:32887)
I20260924 15:22:22.034272 16638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:41067
I20260924 15:22:22.040488 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:22:22.047358 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.041793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53036 (local address 127.31.12.126:46023)
0924 15:22:22.041838 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:22.041840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:22.041917 (+ 77us) server_negotiation.cc:407] Connection header received
0924 15:22:22.041957 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:22.041957 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:22.041964 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:22.042005 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:22.042515 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.043025 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.043717 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.043904 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.043965 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.045043 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:22.045044 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:22.045044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:22.045050 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:22.046469 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:22.046770 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:22.046770 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:22.046770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:22.046815 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:22.046985 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:22.046985 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:22.046985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:22.047103 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:22.047175 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:22.047255 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 15:22:22.047305 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 15:22:22.048709 16638 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53036:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:22:22.050305 16638 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:22:22.056658 17116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.052477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:45091 (local address 127.0.0.1:49252)
0924 15:22:22.052640 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:22.052691 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:22.052757 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:22.053118 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:22.053121 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:22.053123 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:22.053187 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.053188 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.054605 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.054606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.055665 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.055667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.056347 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.056348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.056432 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.056445 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 15:22:22.056538 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 15:22:22.056589 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":3,"thread_start_us":45,"threads_started":1}
I20260924 15:22:22.056666 17120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.052749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49252 (local address 127.31.12.65:45091)
0924 15:22:22.052939 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:22.052943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:22.052955 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:22:22.052998 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:22.053000 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:22.053018 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:22.053100 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:22.053277 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.054495 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.055746 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.056263 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.056347 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.056404 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:22.056556 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 15:22:22.056595 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":71,"threads_started":1}
I20260924 15:22:22.056908 17115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.052519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:32887 (local address 127.0.0.1:37892)
0924 15:22:22.053227 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:22.053237 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:22.053285 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:22.053666 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:22.053667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:22.053669 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:22.053705 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.053707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.054851 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.054852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.055803 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.055805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.056668 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.056668 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.056731 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.056736 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:22:22.056793 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:22:22.056842 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":3,"thread_start_us":49,"threads_started":1}
I20260924 15:22:22.056969 17118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.052715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37892 (local address 127.31.12.67:32887)
0924 15:22:22.053168 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:22.053172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:22.053338 (+ 166us) server_negotiation.cc:407] Connection header received
0924 15:22:22.053380 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:22.053382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:22.053396 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:22.053609 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:22.053783 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.054760 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.055874 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.056527 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.056581 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.056630 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:22.056829 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 15:22:22.056889 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":74,"threads_started":1}
I20260924 15:22:22.057416 17117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.052519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:33129 (local address 127.0.0.1:36186)
0924 15:22:22.052981 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:22.052992 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:22.053057 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:22.053253 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:22.053254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:22.053255 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:22.053313 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.053315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.054389 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.054390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.055728 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.055730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.056666 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.056668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.057242 (+ 574us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.057246 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:22:22.057323 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 15:22:22.057361 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":2,"thread_start_us":50,"threads_started":1}
I20260924 15:22:22.057664 16764 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1a6272dd0542d295becc69eab93ca2 (DEFAULT_TABLE table=test-table [id=a4e60486da0849648e8bbe102b1fa34c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:22:22.057715 17037 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1a6272dd0542d295becc69eab93ca2 (DEFAULT_TABLE table=test-table [id=a4e60486da0849648e8bbe102b1fa34c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:22:22.058019 17037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1a6272dd0542d295becc69eab93ca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:22.058020 16764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1a6272dd0542d295becc69eab93ca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:22.060134 17121 tablet_bootstrap.cc:492] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004: Bootstrap starting.
I20260924 15:22:22.060319 17119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.052734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36186 (local address 127.31.12.66:33129)
0924 15:22:22.052927 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:22.052932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:22.053088 (+ 156us) server_negotiation.cc:407] Connection header received
0924 15:22:22.053135 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:22.053137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:22.053153 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:22.053242 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:22.053404 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.054278 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.055794 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.056508 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.056581 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.056630 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:22.060211 (+ 3581us) server_negotiation.cc:299] Negotiation successful
0924 15:22:22.060254 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":85,"threads_started":1}
I20260924 15:22:22.060843 17121 tablet_bootstrap.cc:654] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:22.061031 16898 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1a6272dd0542d295becc69eab93ca2 (DEFAULT_TABLE table=test-table [id=a4e60486da0849648e8bbe102b1fa34c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:22:22.061291 16898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1a6272dd0542d295becc69eab93ca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:22.061313 17121 log.cc:824] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:22.061637 17122 tablet_bootstrap.cc:492] T 2a1a6272dd0542d295becc69eab93ca2 P bf877393a4094e9c999821c0dfa66e2e: Bootstrap starting.
I20260924 15:22:22.061990 17121 tablet_bootstrap.cc:492] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004: No bootstrap required, opened a new log
I20260924 15:22:22.062068 17121 ts_tablet_manager.cc:1397] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:22:22.063371 17121 raft_consensus.cc:348] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf877393a4094e9c999821c0dfa66e2e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 45091 } } peers { permanent_uuid: "3bf15ee9071a405cb9677d9fb6aabe3a" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33129 } } peers { permanent_uuid: "0187f679583c4cf7a66dcef4e90fe004" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32887 } }
I20260924 15:22:22.063527 17121 raft_consensus.cc:374] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:22.063575 17121 raft_consensus.cc:729] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0187f679583c4cf7a66dcef4e90fe004, State: Initialized, Role: FOLLOWER
I20260924 15:22:22.063725 17121 consensus_queue.cc:260] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf877393a4094e9c999821c0dfa66e2e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 45091 } } peers { permanent_uuid: "3bf15ee9071a405cb9677d9fb6aabe3a" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33129 } } peers { permanent_uuid: "0187f679583c4cf7a66dcef4e90fe004" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32887 } }
I20260924 15:22:22.063767 17124 tablet_bootstrap.cc:492] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a: Bootstrap starting.
I20260924 15:22:22.064014 17121 ts_tablet_manager.cc:1428] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:22:22.064126 17104 heartbeater.cc:503] Master 127.31.12.126:46023 was elected leader, sending a full tablet report...
I20260924 15:22:22.064630 17122 tablet_bootstrap.cc:654] T 2a1a6272dd0542d295becc69eab93ca2 P bf877393a4094e9c999821c0dfa66e2e: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:22.064690 17124 tablet_bootstrap.cc:654] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:22.065253 17122 log.cc:824] T 2a1a6272dd0542d295becc69eab93ca2 P bf877393a4094e9c999821c0dfa66e2e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:22.065351 17124 log.cc:824] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:22.066033 17122 tablet_bootstrap.cc:492] T 2a1a6272dd0542d295becc69eab93ca2 P bf877393a4094e9c999821c0dfa66e2e: No bootstrap required, opened a new log
I20260924 15:22:22.066085 17122 ts_tablet_manager.cc:1397] T 2a1a6272dd0542d295becc69eab93ca2 P bf877393a4094e9c999821c0dfa66e2e: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260924 15:22:22.067351 17122 raft_consensus.cc:348] T 2a1a6272dd0542d295becc69eab93ca2 P bf877393a4094e9c999821c0dfa66e2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf877393a4094e9c999821c0dfa66e2e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 45091 } } peers { permanent_uuid: "3bf15ee9071a405cb9677d9fb6aabe3a" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33129 } } peers { permanent_uuid: "0187f679583c4cf7a66dcef4e90fe004" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32887 } }
I20260924 15:22:22.067476 17122 raft_consensus.cc:374] T 2a1a6272dd0542d295becc69eab93ca2 P bf877393a4094e9c999821c0dfa66e2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:22.067497 17122 raft_consensus.cc:729] T 2a1a6272dd0542d295becc69eab93ca2 P bf877393a4094e9c999821c0dfa66e2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf877393a4094e9c999821c0dfa66e2e, State: Initialized, Role: FOLLOWER
I20260924 15:22:22.067597 17122 consensus_queue.cc:260] T 2a1a6272dd0542d295becc69eab93ca2 P bf877393a4094e9c999821c0dfa66e2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf877393a4094e9c999821c0dfa66e2e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 45091 } } peers { permanent_uuid: "3bf15ee9071a405cb9677d9fb6aabe3a" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33129 } } peers { permanent_uuid: "0187f679583c4cf7a66dcef4e90fe004" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32887 } }
I20260924 15:22:22.067824 17122 ts_tablet_manager.cc:1428] T 2a1a6272dd0542d295becc69eab93ca2 P bf877393a4094e9c999821c0dfa66e2e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:22:22.067823 16830 heartbeater.cc:503] Master 127.31.12.126:46023 was elected leader, sending a full tablet report...
I20260924 15:22:22.068249 17124 tablet_bootstrap.cc:492] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a: No bootstrap required, opened a new log
I20260924 15:22:22.068312 17124 ts_tablet_manager.cc:1397] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 15:22:22.069831 17124 raft_consensus.cc:348] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf877393a4094e9c999821c0dfa66e2e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 45091 } } peers { permanent_uuid: "3bf15ee9071a405cb9677d9fb6aabe3a" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33129 } } peers { permanent_uuid: "0187f679583c4cf7a66dcef4e90fe004" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32887 } }
I20260924 15:22:22.069978 17124 raft_consensus.cc:374] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:22.070000 17124 raft_consensus.cc:729] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bf15ee9071a405cb9677d9fb6aabe3a, State: Initialized, Role: FOLLOWER
I20260924 15:22:22.070098 17124 consensus_queue.cc:260] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf877393a4094e9c999821c0dfa66e2e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 45091 } } peers { permanent_uuid: "3bf15ee9071a405cb9677d9fb6aabe3a" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33129 } } peers { permanent_uuid: "0187f679583c4cf7a66dcef4e90fe004" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32887 } }
I20260924 15:22:22.070305 17124 ts_tablet_manager.cc:1428] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:22:22.070372 16968 heartbeater.cc:503] Master 127.31.12.126:46023 was elected leader, sending a full tablet report...
W20260924 15:22:22.120157 16969 tablet.cc:2369] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:22:22.166384 17125 raft_consensus.cc:482] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:22:22.166513 17125 raft_consensus.cc:504] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf877393a4094e9c999821c0dfa66e2e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 45091 } } peers { permanent_uuid: "3bf15ee9071a405cb9677d9fb6aabe3a" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33129 } } peers { permanent_uuid: "0187f679583c4cf7a66dcef4e90fe004" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32887 } }
I20260924 15:22:22.167230 17125 leader_election.cc:288] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf877393a4094e9c999821c0dfa66e2e (127.31.12.65:45091), 3bf15ee9071a405cb9677d9fb6aabe3a (127.31.12.66:33129)
I20260924 15:22:22.169760 17119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.167249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:35429 (local address 127.31.12.66:33129)
0924 15:22:22.167289 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:22.167291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:22.167313 (+ 22us) server_negotiation.cc:407] Connection header received
0924 15:22:22.167340 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:22.167341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:22.167353 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:22.167431 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:22.167625 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.168085 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.169021 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.169383 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.169489 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.169498 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:22.169659 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 15:22:22.169698 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:22:22.169760 17106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.167023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:33129 (local address 127.31.12.67:35429)
0924 15:22:22.167074 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:22.167087 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:22.167127 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:22.167489 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:22.167490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:22.167491 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:22.167549 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.167551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.168176 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.168177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.168924 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.168926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.169480 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.169481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.169564 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.169567 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:22:22.169653 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 15:22:22.169691 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 15:22:22.169873 17130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.167023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:45091 (local address 127.31.12.67:54251)
0924 15:22:22.167323 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:22.167335 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:22.167418 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:22.167569 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:22.167569 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:22.167571 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:22.167643 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.167644 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.168277 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.168278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.168983 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.168985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.169568 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.169569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.169660 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.169664 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:22:22.169731 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 15:22:22.169771 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":53,"thread_start_us":124,"threads_started":1}
I20260924 15:22:22.169857 17120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.167125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:54251 (local address 127.31.12.65:45091)
0924 15:22:22.167172 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:22.167174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:22.167405 (+ 231us) server_negotiation.cc:407] Connection header received
0924 15:22:22.167481 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:22.167482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:22.167492 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:22.167551 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:22.167709 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.168183 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.169073 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.169485 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.169569 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.169577 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:22.169737 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 15:22:22.169774 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:22:22.170149 16920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1a6272dd0542d295becc69eab93ca2" candidate_uuid: "0187f679583c4cf7a66dcef4e90fe004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bf15ee9071a405cb9677d9fb6aabe3a" is_pre_election: true
I20260924 15:22:22.170271 16920 raft_consensus.cc:2454] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0187f679583c4cf7a66dcef4e90fe004 in term 0.
I20260924 15:22:22.170535 16990 leader_election.cc:302] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0187f679583c4cf7a66dcef4e90fe004, 3bf15ee9071a405cb9677d9fb6aabe3a; no voters:
I20260924 15:22:22.170578 16784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1a6272dd0542d295becc69eab93ca2" candidate_uuid: "0187f679583c4cf7a66dcef4e90fe004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf877393a4094e9c999821c0dfa66e2e" is_pre_election: true
I20260924 15:22:22.170707 16784 raft_consensus.cc:2454] T 2a1a6272dd0542d295becc69eab93ca2 P bf877393a4094e9c999821c0dfa66e2e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0187f679583c4cf7a66dcef4e90fe004 in term 0.
I20260924 15:22:22.170688 17125 raft_consensus.cc:2781] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:22:22.170756 17125 raft_consensus.cc:482] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:22:22.170794 17125 raft_consensus.cc:3037] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:22.171501 17125 raft_consensus.cc:504] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf877393a4094e9c999821c0dfa66e2e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 45091 } } peers { permanent_uuid: "3bf15ee9071a405cb9677d9fb6aabe3a" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33129 } } peers { permanent_uuid: "0187f679583c4cf7a66dcef4e90fe004" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32887 } }
I20260924 15:22:22.171646 17125 leader_election.cc:288] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [CANDIDATE]: Term 1 election: Requested vote from peers bf877393a4094e9c999821c0dfa66e2e (127.31.12.65:45091), 3bf15ee9071a405cb9677d9fb6aabe3a (127.31.12.66:33129)
I20260924 15:22:22.171803 16920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1a6272dd0542d295becc69eab93ca2" candidate_uuid: "0187f679583c4cf7a66dcef4e90fe004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bf15ee9071a405cb9677d9fb6aabe3a"
I20260924 15:22:22.171861 16784 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1a6272dd0542d295becc69eab93ca2" candidate_uuid: "0187f679583c4cf7a66dcef4e90fe004" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf877393a4094e9c999821c0dfa66e2e"
I20260924 15:22:22.171903 16920 raft_consensus.cc:3037] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:22.171947 16784 raft_consensus.cc:3037] T 2a1a6272dd0542d295becc69eab93ca2 P bf877393a4094e9c999821c0dfa66e2e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:22.172624 16784 raft_consensus.cc:2454] T 2a1a6272dd0542d295becc69eab93ca2 P bf877393a4094e9c999821c0dfa66e2e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0187f679583c4cf7a66dcef4e90fe004 in term 1.
I20260924 15:22:22.172662 16920 raft_consensus.cc:2454] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0187f679583c4cf7a66dcef4e90fe004 in term 1.
I20260924 15:22:22.172833 16991 leader_election.cc:302] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0187f679583c4cf7a66dcef4e90fe004, bf877393a4094e9c999821c0dfa66e2e; no voters:
I20260924 15:22:22.172956 17125 raft_consensus.cc:2781] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:22.173193 17125 raft_consensus.cc:686] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [term 1 LEADER]: Becoming Leader. State: Replica: 0187f679583c4cf7a66dcef4e90fe004, State: Running, Role: LEADER
I20260924 15:22:22.173324 17125 consensus_queue.cc:237] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf877393a4094e9c999821c0dfa66e2e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 45091 } } peers { permanent_uuid: "3bf15ee9071a405cb9677d9fb6aabe3a" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33129 } } peers { permanent_uuid: "0187f679583c4cf7a66dcef4e90fe004" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32887 } }
I20260924 15:22:22.174242 16638 catalog_manager.cc:5495] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0187f679583c4cf7a66dcef4e90fe004 (127.31.12.67). New cstate: current_term: 1 leader_uuid: "0187f679583c4cf7a66dcef4e90fe004" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf877393a4094e9c999821c0dfa66e2e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 45091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bf15ee9071a405cb9677d9fb6aabe3a" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0187f679583c4cf7a66dcef4e90fe004" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 32887 } health_report { overall_health: HEALTHY } } }
I20260924 15:22:22.200474 17118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.197996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37902 (local address 127.31.12.67:32887)
0924 15:22:22.198057 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:22.198059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:22.198070 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:22:22.198107 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:22.198108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:22.198125 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:22.198213 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:22:22.198352 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.198917 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.199654 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.199863 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.199946 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.200110 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:22:22.200218 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:22:22.200307 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:22.200332 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 15:22:22.200352 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:22:22.202420 16784 raft_consensus.cc:1261] T 2a1a6272dd0542d295becc69eab93ca2 P bf877393a4094e9c999821c0dfa66e2e [term 1 FOLLOWER]: Refusing update from remote peer 0187f679583c4cf7a66dcef4e90fe004: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:22:22.202493 16920 raft_consensus.cc:1261] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a [term 1 FOLLOWER]: Refusing update from remote peer 0187f679583c4cf7a66dcef4e90fe004: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:22:22.202739 17125 consensus_queue.cc:1035] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf877393a4094e9c999821c0dfa66e2e" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 45091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:22:22.202867 17131 consensus_queue.cc:1035] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bf15ee9071a405cb9677d9fb6aabe3a" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 33129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:22:22.206197 17136 mvcc.cc:205] Tried to move back new op lower bound from 7332918649658322944 to 7332918649542807552. Current Snapshot: MvccSnapshot[applied={T|T < 7332918649658322944}]
I20260924 15:22:22.207248 16637 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:22:22.207350 16637 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53036:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:22:22.208141 16637 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c86e1a817d1846ae8e25bcd03d04c5cc: Sending DeleteTablet for 3 replicas of tablet 2a1a6272dd0542d295becc69eab93ca2
I20260924 15:22:22.208452 17037 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1a6272dd0542d295becc69eab93ca2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:22:22 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:37892
I20260924 15:22:22.208462 16764 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1a6272dd0542d295becc69eab93ca2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:22:22 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:49252
I20260924 15:22:22.208462 16898 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1a6272dd0542d295becc69eab93ca2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:22:22 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:36186
I20260924 15:22:22.208700 17145 tablet_replica.cc:330] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a: stopping tablet replica
I20260924 15:22:22.208742 17143 tablet_replica.cc:330] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004: stopping tablet replica
I20260924 15:22:22.208801 17145 raft_consensus.cc:2229] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:22:22.208943 17145 raft_consensus.cc:2258] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:22:22.208946 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 16700
I20260924 15:22:22.210043 17145 ts_tablet_manager.cc:1905] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:22:22.211655 17145 ts_tablet_manager.cc:1918] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:22:22.211730 17145 log.cc:1196] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/2a1a6272dd0542d295becc69eab93ca2
I20260924 15:22:22.212052 17145 ts_tablet_manager.cc:1939] T 2a1a6272dd0542d295becc69eab93ca2 P 3bf15ee9071a405cb9677d9fb6aabe3a: Deleting consensus metadata
I20260924 15:22:22.212448 16623 catalog_manager.cc:4841] TS 3bf15ee9071a405cb9677d9fb6aabe3a (127.31.12.66:33129): tablet 2a1a6272dd0542d295becc69eab93ca2 (table test-table [id=a4e60486da0849648e8bbe102b1fa34c]) successfully deleted
I20260924 15:22:22.213550 17143 raft_consensus.cc:2229] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:22:22.213665 17143 raft_consensus.cc:2258] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:22:22.214071 17143 ts_tablet_manager.cc:1905] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:22:22.216071 17143 ts_tablet_manager.cc:1918] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:22:22.216148 17143 log.cc:1196] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/2a1a6272dd0542d295becc69eab93ca2
I20260924 15:22:22.216465 17143 ts_tablet_manager.cc:1939] T 2a1a6272dd0542d295becc69eab93ca2 P 0187f679583c4cf7a66dcef4e90fe004: Deleting consensus metadata
W20260924 15:22:22.216838 16624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.12.65:45091 (error 108)
I20260924 15:22:22.216853 16622 catalog_manager.cc:4841] TS 0187f679583c4cf7a66dcef4e90fe004 (127.31.12.67:32887): tablet 2a1a6272dd0542d295becc69eab93ca2 (table test-table [id=a4e60486da0849648e8bbe102b1fa34c]) successfully deleted
I20260924 15:22:22.217332 17117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.217210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:45091 (local address 127.0.0.1:49258)
0924 15:22:22.217255 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:22.217295 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.65:45091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 15:22:22.217510 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 16835
W20260924 15:22:22.217483 16624 catalog_manager.cc:4584] TS bf877393a4094e9c999821c0dfa66e2e (127.31.12.65:45091): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2a1a6272dd0542d295becc69eab93ca2: Network error: Client connection negotiation failed: client connection to 127.31.12.65:45091: connect: Connection refused (error 111)
I20260924 15:22:22.225520 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 16973
I20260924 15:22:22.230233 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 16606
2026-09-24T15:22:22Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2054 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T15:22:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:22Z Disabled control of system clock
I20260924 15:22:22.256541 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:35529
--webserver_interface=127.31.12.126
--webserver_port=0
--builtin_ntp_servers=127.31.12.84:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:35529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:22:22.333027 17150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:22.333227 17150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:22.333247 17150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:22.334362 17150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:22.334403 17150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:22.334415 17150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:22.334429 17150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:22.334440 17150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:22.335685 17150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:35529
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:35529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:22.335865 17150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:22.336088 17150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:22.340616 17156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:22.340621 17158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:22.340626 17155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:22.340762 17150 server_base.cc:1034] running on GCE node
I20260924 15:22:22.341171 17150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:22.341496 17150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:22.342648 17150 hybrid_clock.cc:648] HybridClock initialized: now 1790263342342634 us; error 34 us; skew 500 ppm
I20260924 15:22:22.343809 17150 webserver.cc:460] Webserver started at http://127.31.12.126:37333/ using document root <none> and password file <none>
I20260924 15:22:22.344069 17150 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:22.344115 17150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:22.344244 17150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:22.345260 17150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "2c421743484942e485e704dc8dc5fab6"
format_stamp: "Formatted at 2026-09-24 15:22:22 on dist-test-slave-zwbb"
server_key: "1bd1d79364f77718a23acef18989a664"
server_key_iv: "8839e72ca05ed970dbdf54c98d3895af"
server_key_version: "encryptionkey@0"
I20260924 15:22:22.345655 17150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "2c421743484942e485e704dc8dc5fab6"
format_stamp: "Formatted at 2026-09-24 15:22:22 on dist-test-slave-zwbb"
server_key: "1bd1d79364f77718a23acef18989a664"
server_key_iv: "8839e72ca05ed970dbdf54c98d3895af"
server_key_version: "encryptionkey@0"
I20260924 15:22:22.346872 17150 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:22:22.347551 17164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:22.347728 17150 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:22:22.347793 17150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "2c421743484942e485e704dc8dc5fab6"
format_stamp: "Formatted at 2026-09-24 15:22:22 on dist-test-slave-zwbb"
server_key: "1bd1d79364f77718a23acef18989a664"
server_key_iv: "8839e72ca05ed970dbdf54c98d3895af"
server_key_version: "encryptionkey@0"
I20260924 15:22:22.347877 17150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:22.358026 17150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:22.358299 17150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:22.358453 17150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:22.362254 17150 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:35529
I20260924 15:22:22.362286 17216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:35529 every 8 connection(s)
I20260924 15:22:22.362602 17150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:22.363550 17217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:22.365773 17217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6: Bootstrap starting.
I20260924 15:22:22.366307 17217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:22.366600 17217 log.cc:824] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:22.367233 17217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6: No bootstrap required, opened a new log
I20260924 15:22:22.368757 17217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c421743484942e485e704dc8dc5fab6" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35529 } }
I20260924 15:22:22.368887 17217 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:22.368911 17217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c421743484942e485e704dc8dc5fab6, State: Initialized, Role: FOLLOWER
I20260924 15:22:22.369063 17217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c421743484942e485e704dc8dc5fab6" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35529 } }
I20260924 15:22:22.369122 17217 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:22.369143 17217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:22.369205 17217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:22.369777 17217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c421743484942e485e704dc8dc5fab6" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35529 } }
I20260924 15:22:22.369882 17217 leader_election.cc:302] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c421743484942e485e704dc8dc5fab6; no voters:
I20260924 15:22:22.370064 17217 leader_election.cc:288] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:22.370110 17220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:22.370342 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 17150
I20260924 15:22:22.370478 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:22.370679 31793 external_mini_cluster.cc:1428] Setting key 31fbfdb94edd5d328810e4dba3a38c4e
I20260924 15:22:22.370296 17217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:22.370899 17220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6 [term 1 LEADER]: Becoming Leader. State: Replica: 2c421743484942e485e704dc8dc5fab6, State: Running, Role: LEADER
I20260924 15:22:22.371049 17220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c421743484942e485e704dc8dc5fab6" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35529 } }
I20260924 15:22:22.371551 17222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c421743484942e485e704dc8dc5fab6. Latest consensus state: current_term: 1 leader_uuid: "2c421743484942e485e704dc8dc5fab6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c421743484942e485e704dc8dc5fab6" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35529 } } }
I20260924 15:22:22.371635 17222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:22.372452 17221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c421743484942e485e704dc8dc5fab6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c421743484942e485e704dc8dc5fab6" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35529 } } }
I20260924 15:22:22.372531 17221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6 [sys.catalog]: This master's current role is: LEADER
W20260924 15:22:22.372984 17237 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:22:22.373041 17237 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:22:22.373135 17231 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:22.373297 17231 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:22.374825 17231 catalog_manager.cc:1347] Generated new cluster ID: 84010cf73d0e41339fef1d2465b63d88
I20260924 15:22:22.374878 17231 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:22.377112 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.371299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56840 (local address 127.31.12.126:35529)
0924 15:22:22.371495 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:22.371505 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:22.371586 (+ 81us) server_negotiation.cc:407] Connection header received
0924 15:22:22.371742 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:22.371749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:22.371819 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:22.371969 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:22.372306 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.373728 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.374838 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.375457 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.375547 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.376424 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:22.376428 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:22.376429 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:22.376619 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:22.376657 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:22.376671 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:22.376732 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:22.376849 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 15:22:22.376919 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":66,"threads_started":1}
I20260924 15:22:22.388926 17231 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:22.389374 17231 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:22.394174 17231 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6: Generated new TSK 0
I20260924 15:22:22.394352 17231 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:22.399767 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:35529
--builtin_ntp_servers=127.31.12.84:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:22:22.478039 17242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:22.478209 17242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:22.478229 17242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:22.479269 17242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:22.479303 17242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:22.479324 17242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:22:22.480474 17242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:35529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:22.480686 17242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:22.480965 17242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:22.486590 17248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:22.486824 17242 server_base.cc:1034] running on GCE node
W20260924 15:22:22.486588 17250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:22.486594 17247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:22.487195 17242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:22.487454 17242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:22.488603 17242 hybrid_clock.cc:648] HybridClock initialized: now 1790263342488583 us; error 38 us; skew 500 ppm
I20260924 15:22:22.489818 17242 webserver.cc:460] Webserver started at http://127.31.12.65:46529/ using document root <none> and password file <none>
I20260924 15:22:22.490043 17242 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:22.490105 17242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:22.490243 17242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:22.491228 17242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "7d09db04024245cd8aac8dc8f8d79abd"
format_stamp: "Formatted at 2026-09-24 15:22:22 on dist-test-slave-zwbb"
server_key: "581f53e4ab64209b6a09401e181ede5a"
server_key_iv: "b1962da86305b3058dac64b6e6898f63"
server_key_version: "encryptionkey@0"
I20260924 15:22:22.491568 17242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "7d09db04024245cd8aac8dc8f8d79abd"
format_stamp: "Formatted at 2026-09-24 15:22:22 on dist-test-slave-zwbb"
server_key: "581f53e4ab64209b6a09401e181ede5a"
server_key_iv: "b1962da86305b3058dac64b6e6898f63"
server_key_version: "encryptionkey@0"
I20260924 15:22:22.492821 17242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:22:22.493598 17256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:22.493777 17242 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 15:22:22.493870 17242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "7d09db04024245cd8aac8dc8f8d79abd"
format_stamp: "Formatted at 2026-09-24 15:22:22 on dist-test-slave-zwbb"
server_key: "581f53e4ab64209b6a09401e181ede5a"
server_key_iv: "b1962da86305b3058dac64b6e6898f63"
server_key_version: "encryptionkey@0"
I20260924 15:22:22.493983 17242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:22.501739 17242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:22.502019 17242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:22.502182 17242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:22.502395 17242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:22.502712 17242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:22:22.502775 17242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:22.502816 17242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:22:22.502843 17242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:22.508680 17242 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:39989
I20260924 15:22:22.508723 17369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:39989 every 8 connection(s)
I20260924 15:22:22.509042 17242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:22:22.513950 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.509707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:49669 (local address 127.31.12.126:35529)
0924 15:22:22.509797 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:22.509800 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:22.510016 (+ 216us) server_negotiation.cc:407] Connection header received
0924 15:22:22.510493 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:22.510494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:22.510507 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:22.510561 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:22.511181 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.511677 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.512603 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.512792 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.512847 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.513685 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:22.513686 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:22.513686 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:22.513691 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:22.513706 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:22.513711 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:22.513762 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:22.513858 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 15:22:22.513893 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 15:22:22.514114 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.509689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:35529 (local address 127.31.12.65:49669)
0924 15:22:22.509891 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:22.509927 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:22.510386 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:22.510613 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:22.510616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:22.510624 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:22.511100 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.511104 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.511754 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.511755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.512520 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.512522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.512868 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.512869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.512973 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.513508 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:22:22.513509 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:22.513545 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:22:22.513574 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:22.513779 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:22.513784 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:22.513787 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:22:22.513842 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 15:22:22.513924 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":105,"threads_started":1}
I20260924 15:22:22.514621 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 17242
I20260924 15:22:22.514688 17370 heartbeater.cc:347] Connected to a master server at 127.31.12.126:35529
I20260924 15:22:22.514755 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:22:22.514791 17370 heartbeater.cc:464] Registering TS with master...
I20260924 15:22:22.514955 31793 external_mini_cluster.cc:1428] Setting key 723579ce814e0ab140236a343234f470
I20260924 15:22:22.515024 17370 heartbeater.cc:511] Master 127.31.12.126:35529 requested a full tablet report, sending...
I20260924 15:22:22.515594 17180 ts_manager.cc:194] Registered new tserver with Master: 7d09db04024245cd8aac8dc8f8d79abd (127.31.12.65:39989)
I20260924 15:22:22.515911 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:35529
--builtin_ntp_servers=127.31.12.84:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 15:22:22.516352 17180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.12.65:49669
W20260924 15:22:22.597589 17373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:22.597801 17373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:22.597852 17373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:22.599018 17373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:22.599087 17373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:22.599130 17373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:22:22.600487 17373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:35529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:22.600745 17373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:22.601044 17373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:22:22.606012 17373 server_base.cc:1034] running on GCE node
W20260924 15:22:22.605966 17379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:22.605979 17378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:22.605963 17381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:22.606333 17373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:22.606602 17373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:22.607791 17373 hybrid_clock.cc:648] HybridClock initialized: now 1790263342607778 us; error 25 us; skew 500 ppm
I20260924 15:22:22.608995 17373 webserver.cc:460] Webserver started at http://127.31.12.66:43277/ using document root <none> and password file <none>
I20260924 15:22:22.609263 17373 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:22.609335 17373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:22.609519 17373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:22.610486 17373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "a78417deaaa54ebebdf77fb2a480d128"
format_stamp: "Formatted at 2026-09-24 15:22:22 on dist-test-slave-zwbb"
server_key: "f8e4fa1841bc94291fec066dc5c71e8e"
server_key_iv: "8505e1210744b14fd9736eb40e255b83"
server_key_version: "encryptionkey@0"
I20260924 15:22:22.610816 17373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "a78417deaaa54ebebdf77fb2a480d128"
format_stamp: "Formatted at 2026-09-24 15:22:22 on dist-test-slave-zwbb"
server_key: "f8e4fa1841bc94291fec066dc5c71e8e"
server_key_iv: "8505e1210744b14fd9736eb40e255b83"
server_key_version: "encryptionkey@0"
I20260924 15:22:22.612075 17373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:22:22.612752 17388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:22.612955 17373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:22.613055 17373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "a78417deaaa54ebebdf77fb2a480d128"
format_stamp: "Formatted at 2026-09-24 15:22:22 on dist-test-slave-zwbb"
server_key: "f8e4fa1841bc94291fec066dc5c71e8e"
server_key_iv: "8505e1210744b14fd9736eb40e255b83"
server_key_version: "encryptionkey@0"
I20260924 15:22:22.613143 17373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:22.628818 17373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:22.629101 17373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:22.629243 17373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:22.629496 17373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:22.629814 17373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:22:22.629873 17373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:22.629922 17373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:22:22.629951 17373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:22.635672 17373 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:43279
I20260924 15:22:22.635715 17501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:43279 every 8 connection(s)
I20260924 15:22:22.636036 17373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:22:22.640913 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 17373
I20260924 15:22:22.641053 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:22:22.641292 31793 external_mini_cluster.cc:1428] Setting key d2ced0326b96be0335c62c47efed34a4
I20260924 15:22:22.641327 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.636719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:37429 (local address 127.31.12.126:35529)
0924 15:22:22.636785 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:22.636787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:22.637023 (+ 236us) server_negotiation.cc:407] Connection header received
0924 15:22:22.637638 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:22.637639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:22.637652 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:22.637706 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:22.638396 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.638900 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.639892 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.640103 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.640181 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.641011 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:22.641013 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:22.641013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:22.641018 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:22.641034 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:22.641038 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:22.641090 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:22.641206 (+ 116us) server_negotiation.cc:299] Negotiation successful
0924 15:22:22.641241 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:22:22.641572 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.636691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:35529 (local address 127.31.12.66:37429)
0924 15:22:22.636903 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:22.636937 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:22.637523 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:22.637776 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:22.637780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:22.637789 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:22.638289 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.638294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.638998 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.638999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.639783 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.639785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.640191 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.640192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.640341 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.640875 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:22:22.640875 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:22.640904 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:22:22.640926 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:22.641097 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:22.641102 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:22.641105 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:22:22.641193 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 15:22:22.641281 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":92,"threads_started":1}
I20260924 15:22:22.642091 17502 heartbeater.cc:347] Connected to a master server at 127.31.12.126:35529
I20260924 15:22:22.642190 17502 heartbeater.cc:464] Registering TS with master...
I20260924 15:22:22.642396 17502 heartbeater.cc:511] Master 127.31.12.126:35529 requested a full tablet report, sending...
I20260924 15:22:22.642495 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:35529
--builtin_ntp_servers=127.31.12.84:36985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 15:22:22.642747 17180 ts_manager.cc:194] Registered new tserver with Master: a78417deaaa54ebebdf77fb2a480d128 (127.31.12.66:43279)
I20260924 15:22:22.643224 17180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.12.66:37429
W20260924 15:22:22.725806 17505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:22.726020 17505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:22.726044 17505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:22.727155 17505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:22.727210 17505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:22.727238 17505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:22:22.728528 17505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:36985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:35529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:22.728706 17505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:22.728935 17505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:22.733770 17513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:22.733821 17510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:22.733778 17511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:22.734031 17505 server_base.cc:1034] running on GCE node
I20260924 15:22:22.734251 17505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:22.734499 17505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:22.735668 17505 hybrid_clock.cc:648] HybridClock initialized: now 1790263342735661 us; error 36 us; skew 500 ppm
I20260924 15:22:22.737004 17505 webserver.cc:460] Webserver started at http://127.31.12.67:34181/ using document root <none> and password file <none>
I20260924 15:22:22.737254 17505 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:22.737324 17505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:22.737520 17505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:22.738561 17505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "a53d735e441a44b891fc8acd9f1d9d4e"
format_stamp: "Formatted at 2026-09-24 15:22:22 on dist-test-slave-zwbb"
server_key: "7653ad296b762038f6e41a79a143f175"
server_key_iv: "61a20eeada953df809e7056bb71cb3a5"
server_key_version: "encryptionkey@0"
I20260924 15:22:22.738942 17505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "a53d735e441a44b891fc8acd9f1d9d4e"
format_stamp: "Formatted at 2026-09-24 15:22:22 on dist-test-slave-zwbb"
server_key: "7653ad296b762038f6e41a79a143f175"
server_key_iv: "61a20eeada953df809e7056bb71cb3a5"
server_key_version: "encryptionkey@0"
I20260924 15:22:22.740229 17505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:22:22.740908 17519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:22.741092 17505 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:22.741186 17505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "a53d735e441a44b891fc8acd9f1d9d4e"
format_stamp: "Formatted at 2026-09-24 15:22:22 on dist-test-slave-zwbb"
server_key: "7653ad296b762038f6e41a79a143f175"
server_key_iv: "61a20eeada953df809e7056bb71cb3a5"
server_key_version: "encryptionkey@0"
I20260924 15:22:22.741262 17505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:22.766173 17505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:22.766491 17505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:22.766647 17505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:22.766884 17505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:22.767225 17505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:22:22.767277 17505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:22.767310 17505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:22:22.767335 17505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:22.773305 17505 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:33647
I20260924 15:22:22.773356 17633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:33647 every 8 connection(s)
I20260924 15:22:22.773693 17505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:22:22.777964 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 17505
I20260924 15:22:22.778086 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:22:22.778301 31793 external_mini_cluster.cc:1428] Setting key 5c798703415c0a12dcce30538b69db5f
I20260924 15:22:22.778978 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.774498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:35693 (local address 127.31.12.126:35529)
0924 15:22:22.774550 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:22.774552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:22.774722 (+ 170us) server_negotiation.cc:407] Connection header received
0924 15:22:22.775194 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:22.775195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:22.775205 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:22.775272 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:22.775971 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.776473 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.777419 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.777632 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.777689 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.778625 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:22.778627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:22.778628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:22.778632 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:22.778651 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:22.778656 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:22.778717 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:22.778824 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 15:22:22.778855 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 15:22:22.779171 17636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.774394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:35529 (local address 127.31.12.67:35693)
0924 15:22:22.774585 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:22.774627 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:22.775106 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:22.775352 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:22.775356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:22.775364 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:22.775865 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.775870 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.776555 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.776556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.777328 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.777330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.777717 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.777718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.777833 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.778461 (+ 628us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:22:22.778462 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:22.778506 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:22:22.778538 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:22.778722 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:22.778730 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:22.778734 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:22:22.778817 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 15:22:22.778941 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260924 15:22:22.779655 17634 heartbeater.cc:347] Connected to a master server at 127.31.12.126:35529
I20260924 15:22:22.779768 17634 heartbeater.cc:464] Registering TS with master...
I20260924 15:22:22.779948 17634 heartbeater.cc:511] Master 127.31.12.126:35529 requested a full tablet report, sending...
I20260924 15:22:22.780332 17180 ts_manager.cc:194] Registered new tserver with Master: a53d735e441a44b891fc8acd9f1d9d4e (127.31.12.67:33647)
I20260924 15:22:22.780690 17180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.12.67:35693
I20260924 15:22:22.790457 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:22:22.794716 17228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.791985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56850 (local address 127.31.12.126:35529)
0924 15:22:22.792063 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:22.792065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:22.792158 (+ 93us) server_negotiation.cc:407] Connection header received
0924 15:22:22.792196 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:22.792197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:22.792208 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:22.792259 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:22.792418 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.792887 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.793635 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.793814 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.793866 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.794515 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:22.794521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:22.794522 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:22.794525 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:22.794539 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:22.794542 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:22.794580 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:22.794642 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 15:22:22.794661 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 15:22:22.796176 17180 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:22:22.797928 17180 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:22:22.804333 17646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.800501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:39989 (local address 127.0.0.1:54212)
0924 15:22:22.800613 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:22.800621 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:22.800662 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:22.800957 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:22.800958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:22.800960 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:22.801010 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.801013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.801993 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.801995 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.803315 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.803318 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.804045 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.804046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.804120 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.804133 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 15:22:22.804199 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 15:22:22.804259 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":30,"thread_start_us":65,"threads_started":1}
I20260924 15:22:22.804353 17647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.800612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54212 (local address 127.31.12.65:39989)
0924 15:22:22.800772 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:22.800777 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:22.800787 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:22:22.800833 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:22.800835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:22.800851 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:22.801092 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:22.801156 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.801905 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.803421 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.803954 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.804046 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.804143 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:22.804234 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 15:22:22.804278 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260924 15:22:22.804478 17644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.800329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:33647 (local address 127.0.0.1:52608)
0924 15:22:22.800453 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:22.800461 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:22.800500 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:22.800828 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:22.800831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:22.800833 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:22.800906 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.800908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.802286 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.802287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.803389 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.803391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.804222 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.804222 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.804290 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.804300 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 15:22:22.804331 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 15:22:22.804378 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":37,"thread_start_us":81,"threads_started":1}
I20260924 15:22:22.804741 17645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.800437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.31.12.67:33647)
0924 15:22:22.800633 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:22.800639 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:22.800649 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:22:22.800691 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:22.800693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:22.800710 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:22.800982 (+ 272us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:22.801047 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.802197 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.803475 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.804138 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.804506 (+ 368us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.804602 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:22.804626 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 15:22:22.804664 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
I20260924 15:22:22.804476 17642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.800101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:43279 (local address 127.0.0.1:41340)
0924 15:22:22.800224 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:22.800265 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:22.800323 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:22.800962 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:22.800962 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:22.800964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:22.801011 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.801013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.802332 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.802333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.803278 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.803280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.804218 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.804219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.804290 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.804301 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 15:22:22.804331 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 15:22:22.804378 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"mutex_wait_us":19,"thread_start_us":79,"threads_started":1}
I20260924 15:22:22.805383 17643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.800209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41340 (local address 127.31.12.66:43279)
0924 15:22:22.800787 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:22.800791 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:22.800800 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:22:22.800836 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:22.800838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:22.800851 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:22.801087 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:22.801147 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.802230 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.803363 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.804110 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.805072 (+ 962us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.805243 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:22.805268 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 15:22:22.805308 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":89,"threads_started":1}
I20260924 15:22:22.805578 17568 tablet_service.cc:1467] Processing CreateTablet for tablet 63959062a16f4c4c942a4aa5d3a25856 (DEFAULT_TABLE table=test-table [id=368bf1410d8044c1bfceccb31532bbe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:22:22.805610 17304 tablet_service.cc:1467] Processing CreateTablet for tablet 63959062a16f4c4c942a4aa5d3a25856 (DEFAULT_TABLE table=test-table [id=368bf1410d8044c1bfceccb31532bbe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:22:22.805958 17568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63959062a16f4c4c942a4aa5d3a25856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:22.805964 17304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63959062a16f4c4c942a4aa5d3a25856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:22.805972 17436 tablet_service.cc:1467] Processing CreateTablet for tablet 63959062a16f4c4c942a4aa5d3a25856 (DEFAULT_TABLE table=test-table [id=368bf1410d8044c1bfceccb31532bbe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:22:22.806267 17436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63959062a16f4c4c942a4aa5d3a25856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:22.808727 17648 tablet_bootstrap.cc:492] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e: Bootstrap starting.
I20260924 15:22:22.809264 17649 tablet_bootstrap.cc:492] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd: Bootstrap starting.
I20260924 15:22:22.809499 17648 tablet_bootstrap.cc:654] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:22.809614 17650 tablet_bootstrap.cc:492] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128: Bootstrap starting.
I20260924 15:22:22.809850 17648 log.cc:824] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:22.810022 17649 tablet_bootstrap.cc:654] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:22.810248 17650 tablet_bootstrap.cc:654] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:22.810328 17649 log.cc:824] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:22.810564 17650 log.cc:824] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:22.810653 17648 tablet_bootstrap.cc:492] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e: No bootstrap required, opened a new log
I20260924 15:22:22.810761 17648 ts_tablet_manager.cc:1397] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:22:22.811107 17649 tablet_bootstrap.cc:492] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd: No bootstrap required, opened a new log
I20260924 15:22:22.811193 17649 ts_tablet_manager.cc:1397] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:22:22.811305 17650 tablet_bootstrap.cc:492] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128: No bootstrap required, opened a new log
I20260924 15:22:22.811380 17650 ts_tablet_manager.cc:1397] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:22:22.813213 17649 raft_consensus.cc:348] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78417deaaa54ebebdf77fb2a480d128" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 43279 } } peers { permanent_uuid: "a53d735e441a44b891fc8acd9f1d9d4e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33647 } } peers { permanent_uuid: "7d09db04024245cd8aac8dc8f8d79abd" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 39989 } }
I20260924 15:22:22.813372 17649 raft_consensus.cc:374] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:22.813416 17649 raft_consensus.cc:729] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d09db04024245cd8aac8dc8f8d79abd, State: Initialized, Role: FOLLOWER
I20260924 15:22:22.813555 17649 consensus_queue.cc:260] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78417deaaa54ebebdf77fb2a480d128" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 43279 } } peers { permanent_uuid: "a53d735e441a44b891fc8acd9f1d9d4e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33647 } } peers { permanent_uuid: "7d09db04024245cd8aac8dc8f8d79abd" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 39989 } }
I20260924 15:22:22.813416 17648 raft_consensus.cc:348] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78417deaaa54ebebdf77fb2a480d128" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 43279 } } peers { permanent_uuid: "a53d735e441a44b891fc8acd9f1d9d4e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33647 } } peers { permanent_uuid: "7d09db04024245cd8aac8dc8f8d79abd" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 39989 } }
I20260924 15:22:22.813659 17648 raft_consensus.cc:374] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:22.813702 17648 raft_consensus.cc:729] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a53d735e441a44b891fc8acd9f1d9d4e, State: Initialized, Role: FOLLOWER
I20260924 15:22:22.813835 17649 ts_tablet_manager.cc:1428] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 15:22:22.813829 17648 consensus_queue.cc:260] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78417deaaa54ebebdf77fb2a480d128" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 43279 } } peers { permanent_uuid: "a53d735e441a44b891fc8acd9f1d9d4e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33647 } } peers { permanent_uuid: "7d09db04024245cd8aac8dc8f8d79abd" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 39989 } }
I20260924 15:22:22.813995 17370 heartbeater.cc:503] Master 127.31.12.126:35529 was elected leader, sending a full tablet report...
I20260924 15:22:22.813939 17650 raft_consensus.cc:348] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78417deaaa54ebebdf77fb2a480d128" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 43279 } } peers { permanent_uuid: "a53d735e441a44b891fc8acd9f1d9d4e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33647 } } peers { permanent_uuid: "7d09db04024245cd8aac8dc8f8d79abd" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 39989 } }
I20260924 15:22:22.814079 17650 raft_consensus.cc:374] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:22.814121 17650 raft_consensus.cc:729] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a78417deaaa54ebebdf77fb2a480d128, State: Initialized, Role: FOLLOWER
I20260924 15:22:22.814231 17650 consensus_queue.cc:260] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78417deaaa54ebebdf77fb2a480d128" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 43279 } } peers { permanent_uuid: "a53d735e441a44b891fc8acd9f1d9d4e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33647 } } peers { permanent_uuid: "7d09db04024245cd8aac8dc8f8d79abd" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 39989 } }
I20260924 15:22:22.814378 17648 ts_tablet_manager.cc:1428] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260924 15:22:22.814469 17650 ts_tablet_manager.cc:1428] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:22:22.814504 17634 heartbeater.cc:503] Master 127.31.12.126:35529 was elected leader, sending a full tablet report...
I20260924 15:22:22.814661 17502 heartbeater.cc:503] Master 127.31.12.126:35529 was elected leader, sending a full tablet report...
W20260924 15:22:22.886579 17503 tablet.cc:2369] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:22:22.944145 17654 raft_consensus.cc:482] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:22:22.944274 17654 raft_consensus.cc:504] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78417deaaa54ebebdf77fb2a480d128" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 43279 } } peers { permanent_uuid: "a53d735e441a44b891fc8acd9f1d9d4e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33647 } } peers { permanent_uuid: "7d09db04024245cd8aac8dc8f8d79abd" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 39989 } }
I20260924 15:22:22.944615 17654 leader_election.cc:288] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a78417deaaa54ebebdf77fb2a480d128 (127.31.12.66:43279), a53d735e441a44b891fc8acd9f1d9d4e (127.31.12.67:33647)
I20260924 15:22:22.948556 17643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.944926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:45675 (local address 127.31.12.66:43279)
0924 15:22:22.944992 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:22.944994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:22.945004 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:22:22.945033 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:22.945034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:22.945047 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:22.945110 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:22.945286 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.946089 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.947638 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.948205 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.948274 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.948282 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:22.948442 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 15:22:22.948486 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 15:22:22.948556 17645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.944985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:39225 (local address 127.31.12.67:33647)
0924 15:22:22.945030 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:22.945033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:22.945199 (+ 166us) server_negotiation.cc:407] Connection header received
0924 15:22:22.945252 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:22.945253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:22.945263 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:22.945315 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:22.945467 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.946141 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.947638 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.948188 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.948274 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.948282 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:22.948443 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 15:22:22.948490 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:22:22.948563 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.944771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:43279 (local address 127.31.12.65:45675)
0924 15:22:22.944838 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:22.944863 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:22.944921 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:22.945122 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:22.945122 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:22.945124 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:22.945201 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.945203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.946169 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.946170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.947524 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.947527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.948281 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.948282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.948358 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.948362 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:22:22.948420 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 15:22:22.948471 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":65}
I20260924 15:22:22.948570 17657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.944893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:33647 (local address 127.31.12.65:39225)
0924 15:22:22.945095 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:22.945105 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:22.945160 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:22.945326 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:22.945326 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:22.945328 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:22.945383 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.945384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.946209 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.946209 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.947524 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:22.947527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.948282 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.948282 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:22.948361 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.948363 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:22:22.948427 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 15:22:22.948467 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":3,"thread_start_us":116,"threads_started":1}
I20260924 15:22:22.949011 17588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63959062a16f4c4c942a4aa5d3a25856" candidate_uuid: "7d09db04024245cd8aac8dc8f8d79abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a53d735e441a44b891fc8acd9f1d9d4e" is_pre_election: true
I20260924 15:22:22.949011 17456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63959062a16f4c4c942a4aa5d3a25856" candidate_uuid: "7d09db04024245cd8aac8dc8f8d79abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a78417deaaa54ebebdf77fb2a480d128" is_pre_election: true
I20260924 15:22:22.949174 17588 raft_consensus.cc:2454] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d09db04024245cd8aac8dc8f8d79abd in term 0.
I20260924 15:22:22.949174 17456 raft_consensus.cc:2454] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d09db04024245cd8aac8dc8f8d79abd in term 0.
I20260924 15:22:22.949412 17259 leader_election.cc:302] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d09db04024245cd8aac8dc8f8d79abd, a78417deaaa54ebebdf77fb2a480d128; no voters:
I20260924 15:22:22.949589 17654 raft_consensus.cc:2781] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:22:22.949677 17654 raft_consensus.cc:482] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:22:22.949695 17654 raft_consensus.cc:3037] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:22.950433 17654 raft_consensus.cc:504] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78417deaaa54ebebdf77fb2a480d128" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 43279 } } peers { permanent_uuid: "a53d735e441a44b891fc8acd9f1d9d4e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33647 } } peers { permanent_uuid: "7d09db04024245cd8aac8dc8f8d79abd" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 39989 } }
I20260924 15:22:22.950541 17654 leader_election.cc:288] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd [CANDIDATE]: Term 1 election: Requested vote from peers a78417deaaa54ebebdf77fb2a480d128 (127.31.12.66:43279), a53d735e441a44b891fc8acd9f1d9d4e (127.31.12.67:33647)
I20260924 15:22:22.950744 17456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63959062a16f4c4c942a4aa5d3a25856" candidate_uuid: "7d09db04024245cd8aac8dc8f8d79abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a78417deaaa54ebebdf77fb2a480d128"
I20260924 15:22:22.950754 17588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63959062a16f4c4c942a4aa5d3a25856" candidate_uuid: "7d09db04024245cd8aac8dc8f8d79abd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a53d735e441a44b891fc8acd9f1d9d4e"
I20260924 15:22:22.950840 17456 raft_consensus.cc:3037] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:22.950837 17588 raft_consensus.cc:3037] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:22.951625 17588 raft_consensus.cc:2454] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d09db04024245cd8aac8dc8f8d79abd in term 1.
I20260924 15:22:22.951680 17456 raft_consensus.cc:2454] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d09db04024245cd8aac8dc8f8d79abd in term 1.
I20260924 15:22:22.951860 17259 leader_election.cc:302] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d09db04024245cd8aac8dc8f8d79abd, a78417deaaa54ebebdf77fb2a480d128; no voters:
I20260924 15:22:22.951975 17654 raft_consensus.cc:2781] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:22.952137 17654 raft_consensus.cc:686] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd [term 1 LEADER]: Becoming Leader. State: Replica: 7d09db04024245cd8aac8dc8f8d79abd, State: Running, Role: LEADER
I20260924 15:22:22.952256 17654 consensus_queue.cc:237] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78417deaaa54ebebdf77fb2a480d128" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 43279 } } peers { permanent_uuid: "a53d735e441a44b891fc8acd9f1d9d4e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33647 } } peers { permanent_uuid: "7d09db04024245cd8aac8dc8f8d79abd" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 39989 } }
I20260924 15:22:22.953120 17181 catalog_manager.cc:5495] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d09db04024245cd8aac8dc8f8d79abd (127.31.12.65). New cstate: current_term: 1 leader_uuid: "7d09db04024245cd8aac8dc8f8d79abd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a78417deaaa54ebebdf77fb2a480d128" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 43279 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a53d735e441a44b891fc8acd9f1d9d4e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d09db04024245cd8aac8dc8f8d79abd" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 39989 } health_report { overall_health: HEALTHY } } }
I20260924 15:22:22.984795 17647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:22.982381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54220 (local address 127.31.12.65:39989)
0924 15:22:22.982457 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:22.982459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:22.982467 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:22:22.982514 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:22.982515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:22.982535 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:22.982627 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:22:22.982773 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.983241 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.984007 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:22.984199 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:22.984275 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:22.984440 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:22:22.984540 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:22:22.984639 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:22.984658 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 15:22:22.984678 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 15:22:22.986783 17588 raft_consensus.cc:1261] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e [term 1 FOLLOWER]: Refusing update from remote peer 7d09db04024245cd8aac8dc8f8d79abd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:22:22.986816 17456 raft_consensus.cc:1261] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128 [term 1 FOLLOWER]: Refusing update from remote peer 7d09db04024245cd8aac8dc8f8d79abd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:22:22.987175 17654 consensus_queue.cc:1035] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a53d735e441a44b891fc8acd9f1d9d4e" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 33647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:22:22.987327 17658 consensus_queue.cc:1035] T 63959062a16f4c4c942a4aa5d3a25856 P 7d09db04024245cd8aac8dc8f8d79abd [LEADER]: Connected to new peer: Peer: permanent_uuid: "a78417deaaa54ebebdf77fb2a480d128" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 43279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:22:22.989764 17661 mvcc.cc:205] Tried to move back new op lower bound from 7332918652870983680 to 7332918652733095936. Current Snapshot: MvccSnapshot[applied={T|T < 7332918652870983680}]
I20260924 15:22:22.990077 17662 mvcc.cc:205] Tried to move back new op lower bound from 7332918652870983680 to 7332918652733095936. Current Snapshot: MvccSnapshot[applied={T|T < 7332918652870983680}]
I20260924 15:22:22.992319 17180 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:22:22.992421 17180 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:22:22.993153 17180 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c421743484942e485e704dc8dc5fab6: Sending DeleteTablet for 3 replicas of tablet 63959062a16f4c4c942a4aa5d3a25856
I20260924 15:22:22.993496 17436 tablet_service.cc:1514] Processing DeleteTablet for tablet 63959062a16f4c4c942a4aa5d3a25856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:22:22 UTC) from {username='slave'} at 127.0.0.1:41340
I20260924 15:22:22.993556 17568 tablet_service.cc:1514] Processing DeleteTablet for tablet 63959062a16f4c4c942a4aa5d3a25856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:22:22 UTC) from {username='slave'} at 127.0.0.1:52608
I20260924 15:22:22.993585 17304 tablet_service.cc:1514] Processing DeleteTablet for tablet 63959062a16f4c4c942a4aa5d3a25856 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:22:22 UTC) from {username='slave'} at 127.0.0.1:54212
I20260924 15:22:22.993862 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 17242
I20260924 15:22:22.993950 17671 tablet_replica.cc:330] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128: stopping tablet replica
I20260924 15:22:22.994064 17671 raft_consensus.cc:2229] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:22:22.994179 17671 raft_consensus.cc:2258] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:22:22.994689 17672 tablet_replica.cc:330] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e: stopping tablet replica
I20260924 15:22:22.994726 17671 ts_tablet_manager.cc:1905] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:22:22.994789 17672 raft_consensus.cc:2229] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:22:22.994890 17672 raft_consensus.cc:2258] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:22:22.996047 17672 ts_tablet_manager.cc:1905] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:22:22.996272 17671 ts_tablet_manager.cc:1918] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:22:22.996348 17671 log.cc:1196] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/63959062a16f4c4c942a4aa5d3a25856
I20260924 15:22:22.996634 17671 ts_tablet_manager.cc:1939] T 63959062a16f4c4c942a4aa5d3a25856 P a78417deaaa54ebebdf77fb2a480d128: Deleting consensus metadata
I20260924 15:22:22.997072 17167 catalog_manager.cc:4841] TS a78417deaaa54ebebdf77fb2a480d128 (127.31.12.66:43279): tablet 63959062a16f4c4c942a4aa5d3a25856 (table test-table [id=368bf1410d8044c1bfceccb31532bbe6]) successfully deleted
I20260924 15:22:22.997627 17672 ts_tablet_manager.cc:1918] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:22:22.997700 17672 log.cc:1196] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/63959062a16f4c4c942a4aa5d3a25856
I20260924 15:22:22.997951 17672 ts_tablet_manager.cc:1939] T 63959062a16f4c4c942a4aa5d3a25856 P a53d735e441a44b891fc8acd9f1d9d4e: Deleting consensus metadata
I20260924 15:22:22.998368 17167 catalog_manager.cc:4841] TS a53d735e441a44b891fc8acd9f1d9d4e (127.31.12.67:33647): tablet 63959062a16f4c4c942a4aa5d3a25856 (table test-table [id=368bf1410d8044c1bfceccb31532bbe6]) successfully deleted
W20260924 15:22:23.001060 17167 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.12.65:39989 (error 108)
I20260924 15:22:23.001295 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 17373
I20260924 15:22:23.001561 17642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:23.001415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:39989 (local address 127.0.0.1:54232)
0924 15:22:23.001489 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:23.001525 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.65:39989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 15:22:23.001662 17167 catalog_manager.cc:4584] TS 7d09db04024245cd8aac8dc8f8d79abd (127.31.12.65:39989): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63959062a16f4c4c942a4aa5d3a25856: Network error: Client connection negotiation failed: client connection to 127.31.12.65:39989: connect: Connection refused (error 111)
I20260924 15:22:23.007012 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 17505
I20260924 15:22:23.011861 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 17150
2026-09-24T15:22:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (781 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:23 dist-test-slave-zwbb krb5kdc[17677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:23 dist-test-slave-zwbb krb5kdc[17677](info): set up 2 sockets
Sep 24 15:22:23 dist-test-slave-zwbb krb5kdc[17677](info): commencing operation
krb5kdc: starting...
W20260924 15:22:24.050720 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:24 dist-test-slave-zwbb krb5kdc[17677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:22:24.167806 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:39451
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:39451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:24.246354 17690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:24.246559 17690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:24.246593 17690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:24.247671 17690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:24.247735 17690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:24.247756 17690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:24.247784 17690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:24.247809 17690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:24.249215 17690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:39451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:39451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:24.249496 17690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:24.249725 17690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:24.254261 17696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:24.254261 17698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:24.254343 17695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:24.254477 17690 server_base.cc:1034] running on GCE node
I20260924 15:22:24.254783 17690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:24.255064 17690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:24.256213 17690 hybrid_clock.cc:648] HybridClock initialized: now 1790263344256195 us; error 36 us; skew 500 ppm
Sep 24 15:22:24 dist-test-slave-zwbb krb5kdc[17677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263344, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:24.258710 17690 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:22:24.259191 17690 webserver.cc:460] Webserver started at http://127.31.12.126:42873/ using document root <none> and password file <none>
I20260924 15:22:24.259404 17690 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:24.259449 17690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:24.259636 17690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:24.260674 17690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "25b68c90e7bf44dcbf5d93553c1b63db"
format_stamp: "Formatted at 2026-09-24 15:22:24 on dist-test-slave-zwbb"
server_key: "98ff9c3ab126d9530196a736b03cf25a"
server_key_iv: "d1cdfe92acbf207967d39ba9f708761a"
server_key_version: "encryptionkey@0"
I20260924 15:22:24.261041 17690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "25b68c90e7bf44dcbf5d93553c1b63db"
format_stamp: "Formatted at 2026-09-24 15:22:24 on dist-test-slave-zwbb"
server_key: "98ff9c3ab126d9530196a736b03cf25a"
server_key_iv: "d1cdfe92acbf207967d39ba9f708761a"
server_key_version: "encryptionkey@0"
I20260924 15:22:24.262727 17690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:22:24.263463 17705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:24.263638 17690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:24.263711 17690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "25b68c90e7bf44dcbf5d93553c1b63db"
format_stamp: "Formatted at 2026-09-24 15:22:24 on dist-test-slave-zwbb"
server_key: "98ff9c3ab126d9530196a736b03cf25a"
server_key_iv: "d1cdfe92acbf207967d39ba9f708761a"
server_key_version: "encryptionkey@0"
I20260924 15:22:24.263799 17690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:24.281059 17690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:24.282963 17690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:24.283118 17690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:24.287199 17690 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:39451
I20260924 15:22:24.287340 17758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:39451 every 8 connection(s)
I20260924 15:22:24.287603 17690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:24.288479 17759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:24.290808 17759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db: Bootstrap starting.
I20260924 15:22:24.291415 17759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:24.291709 17759 log.cc:824] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:24.292265 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 17690
I20260924 15:22:24.292348 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:24.292397 17759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db: No bootstrap required, opened a new log
I20260924 15:22:24.292577 31793 external_mini_cluster.cc:1428] Setting key b2d5b6109b0cf3792bbc8d1c9a16d870
I20260924 15:22:24.294891 17759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b68c90e7bf44dcbf5d93553c1b63db" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 39451 } }
I20260924 15:22:24.295038 17759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:24.295087 17759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25b68c90e7bf44dcbf5d93553c1b63db, State: Initialized, Role: FOLLOWER
I20260924 15:22:24.295214 17759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b68c90e7bf44dcbf5d93553c1b63db" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 39451 } }
I20260924 15:22:24.295287 17759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:24.295329 17759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:24.295374 17759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:24.295876 17759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b68c90e7bf44dcbf5d93553c1b63db" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 39451 } }
I20260924 15:22:24.296010 17759 leader_election.cc:302] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25b68c90e7bf44dcbf5d93553c1b63db; no voters:
I20260924 15:22:24.296187 17759 leader_election.cc:288] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:24.296221 17764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:24.296368 17764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db [term 1 LEADER]: Becoming Leader. State: Replica: 25b68c90e7bf44dcbf5d93553c1b63db, State: Running, Role: LEADER
I20260924 15:22:24.296427 17759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:24.296500 17764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b68c90e7bf44dcbf5d93553c1b63db" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 39451 } }
I20260924 15:22:24.296861 17764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25b68c90e7bf44dcbf5d93553c1b63db. Latest consensus state: current_term: 1 leader_uuid: "25b68c90e7bf44dcbf5d93553c1b63db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b68c90e7bf44dcbf5d93553c1b63db" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 39451 } } }
I20260924 15:22:24.296868 17766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25b68c90e7bf44dcbf5d93553c1b63db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25b68c90e7bf44dcbf5d93553c1b63db" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 39451 } } }
I20260924 15:22:24.297019 17764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:24.297031 17766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:24.297734 17772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:24.298334 17772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:24.300051 17772 catalog_manager.cc:1347] Generated new cluster ID: 600761fca9ef4a5abcd55f53e58bffdc
I20260924 15:22:24.300107 17772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:22:24 dist-test-slave-zwbb krb5kdc[17677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:22:24.304937 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:24.293102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40642 (local address 127.31.12.126:39451)
0924 15:22:24.293272 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:24.293279 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:24.293304 (+ 25us) server_negotiation.cc:407] Connection header received
0924 15:22:24.293795 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:24.293802 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:24.293855 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:24.293968 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:24.294803 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.296089 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.297198 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.299005 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.299094 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:24.301759 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:24.301763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:24.301763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:24.301774 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:24.303536 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:24.303850 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:24.303852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:24.303853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:24.303959 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:24.304125 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:24.304125 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:24.304126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:24.304395 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:24.304519 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:24.304592 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 15:22:24.304710 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260924 15:22:24.331331 17772 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:24.332034 17772 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:24.338348 17772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db: Generated new TSK 0
I20260924 15:22:24.338511 17772 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
I20260924 15:22:24.367906 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:39451
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:24.447974 17787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:24.448154 17787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:24.448176 17787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:24.449240 17787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:24.449278 17787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:24.449301 17787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:22:24.450685 17787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:39451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:24.450938 17787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:24.451153 17787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:24.456069 17792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:24.456050 17793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:24.456156 17787 server_base.cc:1034] running on GCE node
W20260924 15:22:24.456051 17795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:24.456472 17787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:24.456699 17787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:24.457854 17787 hybrid_clock.cc:648] HybridClock initialized: now 1790263344457846 us; error 41 us; skew 500 ppm
Sep 24 15:22:24 dist-test-slave-zwbb krb5kdc[17677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263344, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:24.460325 17787 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:22:24.460688 17787 webserver.cc:460] Webserver started at http://127.31.12.65:40367/ using document root <none> and password file <none>
I20260924 15:22:24.460843 17787 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:24.460880 17787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:24.460963 17787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:24.461915 17787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "5f0ffac03be84e5bb3ca54739a6348b3"
format_stamp: "Formatted at 2026-09-24 15:22:24 on dist-test-slave-zwbb"
server_key: "8ad083ff8a909d9f7995e07f92a91d6a"
server_key_iv: "217631ed86d10fbc14f6191169ee67c5"
server_key_version: "encryptionkey@0"
I20260924 15:22:24.462196 17787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "5f0ffac03be84e5bb3ca54739a6348b3"
format_stamp: "Formatted at 2026-09-24 15:22:24 on dist-test-slave-zwbb"
server_key: "8ad083ff8a909d9f7995e07f92a91d6a"
server_key_iv: "217631ed86d10fbc14f6191169ee67c5"
server_key_version: "encryptionkey@0"
I20260924 15:22:24.463445 17787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:22:24.464119 17802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:24.464336 17787 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:22:24.464423 17787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "5f0ffac03be84e5bb3ca54739a6348b3"
format_stamp: "Formatted at 2026-09-24 15:22:24 on dist-test-slave-zwbb"
server_key: "8ad083ff8a909d9f7995e07f92a91d6a"
server_key_iv: "217631ed86d10fbc14f6191169ee67c5"
server_key_version: "encryptionkey@0"
I20260924 15:22:24.464478 17787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:24.481696 17787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:24.483512 17787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:24.483626 17787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:24.483803 17787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:24.484118 17787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:22:24.484151 17787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:24.484194 17787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:22:24.484225 17787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:24.489809 17787 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:36777
I20260924 15:22:24.489882 17918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:36777 every 8 connection(s)
I20260924 15:22:24.490190 17787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:22:24.492864 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 17787
I20260924 15:22:24.492956 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:22:24.493187 31793 external_mini_cluster.cc:1428] Setting key a0faa9d5a0bab7b553bfca55b8833740
Sep 24 15:22:24 dist-test-slave-zwbb krb5kdc[17677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263344, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:22:24.500027 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:24.490883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:35459 (local address 127.31.12.126:39451)
0924 15:22:24.490929 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:24.490932 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:24.491160 (+ 228us) server_negotiation.cc:407] Connection header received
0924 15:22:24.491612 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:24.491613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:24.491623 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:24.491673 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:24.492538 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.493116 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.494589 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.494785 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.494846 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:24.496922 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:24.496924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:24.496924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:24.496932 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:24.498664 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:24.499101 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:24.499103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:24.499103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:24.499166 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:24.499378 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:24.499379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:24.499379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:24.499534 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:24.499609 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:24.499837 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 15:22:24.499927 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:22:24.500056 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:24.490870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:39451 (local address 127.31.12.65:35459)
0924 15:22:24.491042 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:24.491077 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:24.491534 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:24.491724 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:24.491727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:24.492272 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:24.492465 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:24.492469 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.493204 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.493205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:24.494478 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:24.494481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.494873 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.494874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:24.495017 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:24.495471 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:24.495480 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:24.496809 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:24.498740 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:24.498747 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:24.498748 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:24.499029 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:24.499235 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:24.499236 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:24.499236 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:24.499318 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:24.499603 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:24.499608 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:24.499711 (+ 103us) client_negotiation.cc:769] Sending connection context
0924 15:22:24.499788 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 15:22:24.499898 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
I20260924 15:22:24.500591 17919 heartbeater.cc:347] Connected to a master server at 127.31.12.126:39451
I20260924 15:22:24.500685 17919 heartbeater.cc:464] Registering TS with master...
I20260924 15:22:24.500857 17919 heartbeater.cc:511] Master 127.31.12.126:39451 requested a full tablet report, sending...
I20260924 15:22:24.501410 17722 ts_manager.cc:194] Registered new tserver with Master: 5f0ffac03be84e5bb3ca54739a6348b3 (127.31.12.65:36777)
WARNING: no policy specified for kudu/127.31.12.66@KRBTEST.COM; defaulting to no policy
I20260924 15:22:24.502359 17722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:35459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
I20260924 15:22:24.520823 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:39451
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:24.602896 17924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:24.603091 17924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:24.603114 17924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:24.604177 17924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:24.604212 17924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:24.604236 17924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:22:24.605651 17924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:39451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:24.605823 17924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:24.605993 17924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:24.610857 17929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:24.610867 17930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:24.610868 17932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:24.611080 17924 server_base.cc:1034] running on GCE node
I20260924 15:22:24.611322 17924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:24.611563 17924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:24.612721 17924 hybrid_clock.cc:648] HybridClock initialized: now 1790263344612694 us; error 38 us; skew 500 ppm
Sep 24 15:22:24 dist-test-slave-zwbb krb5kdc[17677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263344, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:24.615080 17924 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:22:24.615514 17924 webserver.cc:460] Webserver started at http://127.31.12.66:37149/ using document root <none> and password file <none>
I20260924 15:22:24.615749 17924 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:24.615813 17924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:24.615983 17924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:24.616887 17924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "ae8d062d1fa847079b104eb3b54b659e"
format_stamp: "Formatted at 2026-09-24 15:22:24 on dist-test-slave-zwbb"
server_key: "6fc4294d5f9a9919fedde54b588f4a24"
server_key_iv: "b786d933714ac19174a9dfbe9d6eefae"
server_key_version: "encryptionkey@0"
I20260924 15:22:24.617210 17924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "ae8d062d1fa847079b104eb3b54b659e"
format_stamp: "Formatted at 2026-09-24 15:22:24 on dist-test-slave-zwbb"
server_key: "6fc4294d5f9a9919fedde54b588f4a24"
server_key_iv: "b786d933714ac19174a9dfbe9d6eefae"
server_key_version: "encryptionkey@0"
I20260924 15:22:24.618479 17924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:22:24.619197 17939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:24.619361 17924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:22:24.619462 17924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "ae8d062d1fa847079b104eb3b54b659e"
format_stamp: "Formatted at 2026-09-24 15:22:24 on dist-test-slave-zwbb"
server_key: "6fc4294d5f9a9919fedde54b588f4a24"
server_key_iv: "b786d933714ac19174a9dfbe9d6eefae"
server_key_version: "encryptionkey@0"
I20260924 15:22:24.619546 17924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:24.638953 17924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:24.640585 17924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:24.640733 17924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:24.640944 17924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:24.641225 17924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:22:24.641276 17924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:24.641315 17924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:22:24.641342 17924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:24.647121 17924 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:38811
I20260924 15:22:24.647166 18054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:38811 every 8 connection(s)
I20260924 15:22:24.647470 17924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:22:24 dist-test-slave-zwbb krb5kdc[17677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263344, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:22:24.655668 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:24.648183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:53995 (local address 127.31.12.126:39451)
0924 15:22:24.648252 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:24.648254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:24.648423 (+ 169us) server_negotiation.cc:407] Connection header received
0924 15:22:24.648868 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:24.648869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:24.648878 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:24.648932 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:24.649873 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.650342 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.651251 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.651447 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.651498 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:24.653396 (+ 1898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:24.653397 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:24.653397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:24.653404 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:24.654594 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:24.654925 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:24.654926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:24.654926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:24.654972 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:24.655165 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:24.655166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:24.655166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:24.655275 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:24.655323 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:24.655551 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 15:22:24.655606 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 15:22:24.655855 18057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:24.648146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:39451 (local address 127.31.12.66:53995)
0924 15:22:24.648290 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:24.648331 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:24.648790 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:24.648982 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:24.648985 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:24.649609 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:24.649793 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:24.649799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.650419 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.650420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:24.651181 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:24.651182 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.651518 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.651519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:24.651611 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:24.652038 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:24.652045 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:24.653316 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:24.654658 (+ 1342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:24.654664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:24.654665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:24.654865 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:24.655027 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:24.655027 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:24.655027 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:24.655115 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:24.655327 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:24.655331 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:24.655445 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 15:22:24.655522 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 15:22:24.655630 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260924 15:22:24.655941 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 17924
I20260924 15:22:24.656050 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:22:24.656251 31793 external_mini_cluster.cc:1428] Setting key 45ee036775b0b333d4f7cf6172a5600e
I20260924 15:22:24.656517 18055 heartbeater.cc:347] Connected to a master server at 127.31.12.126:39451
I20260924 15:22:24.656615 18055 heartbeater.cc:464] Registering TS with master...
I20260924 15:22:24.656795 18055 heartbeater.cc:511] Master 127.31.12.126:39451 requested a full tablet report, sending...
I20260924 15:22:24.657224 17722 ts_manager.cc:194] Registered new tserver with Master: ae8d062d1fa847079b104eb3b54b659e (127.31.12.66:38811)
I20260924 15:22:24.657688 17722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:53995
WARNING: no policy specified for kudu/127.31.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
I20260924 15:22:24.680917 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:39451
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:24.761773 18060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:24.761969 18060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:24.761992 18060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:24.763064 18060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:24.763099 18060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:24.763125 18060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:22:24.764523 18060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:39451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:24.764725 18060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:24.764916 18060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:24.770109 18065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:24.770144 18066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:24.770212 18060 server_base.cc:1034] running on GCE node
W20260924 15:22:24.770179 18068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:24.770490 18060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:24.770737 18060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:24.771924 18060 hybrid_clock.cc:648] HybridClock initialized: now 1790263344771901 us; error 39 us; skew 500 ppm
Sep 24 15:22:24 dist-test-slave-zwbb krb5kdc[17677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263344, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:24.774500 18060 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:22:24.774922 18060 webserver.cc:460] Webserver started at http://127.31.12.67:42997/ using document root <none> and password file <none>
I20260924 15:22:24.775151 18060 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:24.775195 18060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:24.775345 18060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:24.776332 18060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "ce88ae5c318847308daeea01af29316d"
format_stamp: "Formatted at 2026-09-24 15:22:24 on dist-test-slave-zwbb"
server_key: "3fb0be83b46eb59afaf922205e265197"
server_key_iv: "e4b7418bfe111b1a180e276918fd5fde"
server_key_version: "encryptionkey@0"
I20260924 15:22:24.776680 18060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "ce88ae5c318847308daeea01af29316d"
format_stamp: "Formatted at 2026-09-24 15:22:24 on dist-test-slave-zwbb"
server_key: "3fb0be83b46eb59afaf922205e265197"
server_key_iv: "e4b7418bfe111b1a180e276918fd5fde"
server_key_version: "encryptionkey@0"
I20260924 15:22:24.778047 18060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:22:24.778736 18075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:24.778920 18060 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:24.778990 18060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "ce88ae5c318847308daeea01af29316d"
format_stamp: "Formatted at 2026-09-24 15:22:24 on dist-test-slave-zwbb"
server_key: "3fb0be83b46eb59afaf922205e265197"
server_key_iv: "e4b7418bfe111b1a180e276918fd5fde"
server_key_version: "encryptionkey@0"
I20260924 15:22:24.779075 18060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:24.787398 18060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:24.789505 18060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:24.789654 18060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:24.789858 18060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:24.790158 18060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:22:24.790210 18060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:24.790248 18060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:22:24.790274 18060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:24.796082 18060 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:35521
I20260924 15:22:24.796149 18190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:35521 every 8 connection(s)
I20260924 15:22:24.796442 18060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
Sep 24 15:22:24 dist-test-slave-zwbb krb5kdc[17677](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263344, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:22:24.805081 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:24.797169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:45979 (local address 127.31.12.126:39451)
0924 15:22:24.797234 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:24.797236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:24.797412 (+ 176us) server_negotiation.cc:407] Connection header received
0924 15:22:24.797896 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:24.797898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:24.797909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:24.797971 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:24.798935 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.799423 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.800359 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.800571 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.800653 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:24.802521 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:24.802522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:24.802523 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:24.802529 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:24.803880 (+ 1351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:24.804244 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:24.804245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:24.804245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:24.804292 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:24.804494 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:24.804494 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:24.804495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:24.804587 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:24.804635 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:24.804897 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 15:22:24.804995 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 15:22:24.805244 18193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:24.797089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:39451 (local address 127.31.12.67:45979)
0924 15:22:24.797288 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:24.797329 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:24.797800 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:24.798032 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:24.798036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:24.798655 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:24.798824 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:24.798828 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.799515 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.799516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:24.800268 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:24.800270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.800665 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.800666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:24.800779 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:24.801151 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:24.801159 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:24.802435 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:24.803967 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:24.803972 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:24.803973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:24.804182 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:24.804347 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:24.804347 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:24.804348 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:24.804440 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:24.804639 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:24.804643 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:24.804741 (+ 98us) client_negotiation.cc:769] Sending connection context
0924 15:22:24.804862 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 15:22:24.805024 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260924 15:22:24.805717 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 18060
I20260924 15:22:24.805738 18191 heartbeater.cc:347] Connected to a master server at 127.31.12.126:39451
I20260924 15:22:24.805823 18191 heartbeater.cc:464] Registering TS with master...
I20260924 15:22:24.805821 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:22:24.806007 18191 heartbeater.cc:511] Master 127.31.12.126:39451 requested a full tablet report, sending...
I20260924 15:22:24.806036 31793 external_mini_cluster.cc:1428] Setting key 159a94a99e449fb0d0d3080a740c7bbd
I20260924 15:22:24.806435 17721 ts_manager.cc:194] Registered new tserver with Master: ce88ae5c318847308daeea01af29316d (127.31.12.67:35521)
I20260924 15:22:24.806880 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:22:24.806895 17721 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:45979
I20260924 15:22:24.814481 17763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:24.808426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.31.12.126:39451)
0924 15:22:24.808493 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:24.808495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:24.808568 (+ 73us) server_negotiation.cc:407] Connection header received
0924 15:22:24.808643 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:24.808643 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:24.808653 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:24.808708 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:24.809270 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.809812 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.810555 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.810755 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.810828 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:24.812039 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:24.812040 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:24.812041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:24.812048 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:24.813202 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:24.813663 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:24.813664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:24.813664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:24.813725 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:24.813949 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:24.813950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:24.813950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:24.814087 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:24.814203 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:24.814325 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 15:22:24.814390 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:22:24.816248 17721 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:22:24.817855 17721 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:22:24.824414 18201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:24.819919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:35521 (local address 127.0.0.1:46250)
0924 15:22:24.820257 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:24.820298 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:24.820395 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:24.820702 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:24.820705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:24.820708 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:24.820792 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:24.820794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.821934 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.821935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:24.823330 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:24.823333 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.824068 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.824069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:24.824162 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:24.824178 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 15:22:24.824261 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 15:22:24.824325 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":2,"thread_start_us":55,"threads_started":1}
I20260924 15:22:24.824486 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:24.819974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:38811 (local address 127.0.0.1:38900)
0924 15:22:24.820507 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:24.820516 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:24.820566 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:24.820731 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:24.820732 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:24.820733 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:24.820795 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:24.820796 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.822196 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.822197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:24.823297 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:24.823299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.824225 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.824226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:24.824285 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:24.824288 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:22:24.824337 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 15:22:24.824379 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":52,"thread_start_us":58,"threads_started":1}
I20260924 15:22:24.824504 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:24.820203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38900 (local address 127.31.12.66:38811)
0924 15:22:24.820385 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:24.820389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:24.820581 (+ 192us) server_negotiation.cc:407] Connection header received
0924 15:22:24.820629 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:24.820630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:24.820644 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:24.820721 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:24.820883 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.822062 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.823381 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.824139 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.824221 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:24.824280 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:24.824352 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 15:22:24.824393 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":83,"threads_started":1}
I20260924 15:22:24.824764 18202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:24.820074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46250 (local address 127.31.12.67:35521)
0924 15:22:24.820231 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:24.820237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:24.820482 (+ 245us) server_negotiation.cc:407] Connection header received
0924 15:22:24.820515 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:24.820517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:24.820528 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:24.820612 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:24.820861 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.821797 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.823440 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.823973 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.824066 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:24.824128 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:24.824643 (+ 515us) server_negotiation.cc:299] Negotiation successful
0924 15:22:24.824688 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260924 15:22:24.824780 18203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:24.820114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54350 (local address 127.31.12.65:36777)
0924 15:22:24.820410 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:24.820415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:24.820426 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:22:24.820501 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:24.820504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:24.820522 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:24.820598 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:24.820903 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.822076 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.823514 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.824205 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.824263 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:24.824309 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:24.824661 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 15:22:24.824706 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":84,"threads_started":1}
I20260924 15:22:24.825357 18199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:24.819919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:36777 (local address 127.0.0.1:54350)
0924 15:22:24.820318 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:24.820330 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:24.820395 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:24.820713 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:24.820714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:24.820716 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:24.820793 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:24.820795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.822196 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.822197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:24.823434 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:24.823437 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:24.824345 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:24.824346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:24.824409 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:24.824413 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:22:24.824479 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 15:22:24.825289 (+ 810us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":68,"thread_start_us":62,"threads_started":1}
I20260924 15:22:24.825249 17988 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1cd873543944caaa403a0238c90419 (DEFAULT_TABLE table=test-table [id=6373cc8fc1b846c4af2ce7bff493c0ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:22:24.825367 18124 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1cd873543944caaa403a0238c90419 (DEFAULT_TABLE table=test-table [id=6373cc8fc1b846c4af2ce7bff493c0ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:22:24.825662 17988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1cd873543944caaa403a0238c90419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:24.825744 18124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1cd873543944caaa403a0238c90419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:24.826023 17851 tablet_service.cc:1467] Processing CreateTablet for tablet 7d1cd873543944caaa403a0238c90419 (DEFAULT_TABLE table=test-table [id=6373cc8fc1b846c4af2ce7bff493c0ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:22:24.826331 17851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d1cd873543944caaa403a0238c90419. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:24.828341 18205 tablet_bootstrap.cc:492] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d: Bootstrap starting.
I20260924 15:22:24.828996 18206 tablet_bootstrap.cc:492] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e: Bootstrap starting.
I20260924 15:22:24.829046 18205 tablet_bootstrap.cc:654] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:24.829679 18207 tablet_bootstrap.cc:492] T 7d1cd873543944caaa403a0238c90419 P 5f0ffac03be84e5bb3ca54739a6348b3: Bootstrap starting.
I20260924 15:22:24.829728 18206 tablet_bootstrap.cc:654] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:24.829751 18205 log.cc:824] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:24.830268 18206 log.cc:824] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:24.830297 18207 tablet_bootstrap.cc:654] T 7d1cd873543944caaa403a0238c90419 P 5f0ffac03be84e5bb3ca54739a6348b3: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:24.830608 18205 tablet_bootstrap.cc:492] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d: No bootstrap required, opened a new log
I20260924 15:22:24.830694 18205 ts_tablet_manager.cc:1397] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:22:24.830863 18207 log.cc:824] T 7d1cd873543944caaa403a0238c90419 P 5f0ffac03be84e5bb3ca54739a6348b3: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:24.831084 18206 tablet_bootstrap.cc:492] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e: No bootstrap required, opened a new log
I20260924 15:22:24.831147 18206 ts_tablet_manager.cc:1397] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:22:24.831616 18207 tablet_bootstrap.cc:492] T 7d1cd873543944caaa403a0238c90419 P 5f0ffac03be84e5bb3ca54739a6348b3: No bootstrap required, opened a new log
I20260924 15:22:24.831750 18207 ts_tablet_manager.cc:1397] T 7d1cd873543944caaa403a0238c90419 P 5f0ffac03be84e5bb3ca54739a6348b3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:22:24.832674 18205 raft_consensus.cc:348] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0ffac03be84e5bb3ca54739a6348b3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 36777 } } peers { permanent_uuid: "ce88ae5c318847308daeea01af29316d" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 35521 } } peers { permanent_uuid: "ae8d062d1fa847079b104eb3b54b659e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 38811 } }
I20260924 15:22:24.833001 18205 raft_consensus.cc:374] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:24.833058 18205 raft_consensus.cc:729] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce88ae5c318847308daeea01af29316d, State: Initialized, Role: FOLLOWER
I20260924 15:22:24.833195 18205 consensus_queue.cc:260] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0ffac03be84e5bb3ca54739a6348b3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 36777 } } peers { permanent_uuid: "ce88ae5c318847308daeea01af29316d" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 35521 } } peers { permanent_uuid: "ae8d062d1fa847079b104eb3b54b659e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 38811 } }
I20260924 15:22:24.833225 18206 raft_consensus.cc:348] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0ffac03be84e5bb3ca54739a6348b3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 36777 } } peers { permanent_uuid: "ce88ae5c318847308daeea01af29316d" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 35521 } } peers { permanent_uuid: "ae8d062d1fa847079b104eb3b54b659e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 38811 } }
I20260924 15:22:24.833417 18206 raft_consensus.cc:374] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:24.833498 18206 raft_consensus.cc:729] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8d062d1fa847079b104eb3b54b659e, State: Initialized, Role: FOLLOWER
I20260924 15:22:24.833513 18205 ts_tablet_manager.cc:1428] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:22:24.833630 18191 heartbeater.cc:503] Master 127.31.12.126:39451 was elected leader, sending a full tablet report...
I20260924 15:22:24.833519 18207 raft_consensus.cc:348] T 7d1cd873543944caaa403a0238c90419 P 5f0ffac03be84e5bb3ca54739a6348b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0ffac03be84e5bb3ca54739a6348b3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 36777 } } peers { permanent_uuid: "ce88ae5c318847308daeea01af29316d" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 35521 } } peers { permanent_uuid: "ae8d062d1fa847079b104eb3b54b659e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 38811 } }
I20260924 15:22:24.833621 18206 consensus_queue.cc:260] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0ffac03be84e5bb3ca54739a6348b3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 36777 } } peers { permanent_uuid: "ce88ae5c318847308daeea01af29316d" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 35521 } } peers { permanent_uuid: "ae8d062d1fa847079b104eb3b54b659e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 38811 } }
I20260924 15:22:24.833683 18207 raft_consensus.cc:374] T 7d1cd873543944caaa403a0238c90419 P 5f0ffac03be84e5bb3ca54739a6348b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:24.833716 18207 raft_consensus.cc:729] T 7d1cd873543944caaa403a0238c90419 P 5f0ffac03be84e5bb3ca54739a6348b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f0ffac03be84e5bb3ca54739a6348b3, State: Initialized, Role: FOLLOWER
I20260924 15:22:24.833833 18207 consensus_queue.cc:260] T 7d1cd873543944caaa403a0238c90419 P 5f0ffac03be84e5bb3ca54739a6348b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0ffac03be84e5bb3ca54739a6348b3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 36777 } } peers { permanent_uuid: "ce88ae5c318847308daeea01af29316d" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 35521 } } peers { permanent_uuid: "ae8d062d1fa847079b104eb3b54b659e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 38811 } }
I20260924 15:22:24.833894 18206 ts_tablet_manager.cc:1428] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:22:24.833992 18055 heartbeater.cc:503] Master 127.31.12.126:39451 was elected leader, sending a full tablet report...
I20260924 15:22:24.834097 18207 ts_tablet_manager.cc:1428] T 7d1cd873543944caaa403a0238c90419 P 5f0ffac03be84e5bb3ca54739a6348b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:22:24.834254 17919 heartbeater.cc:503] Master 127.31.12.126:39451 was elected leader, sending a full tablet report...
W20260924 15:22:24.898005 18056 tablet.cc:2369] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:22:24.990948 17920 tablet.cc:2369] T 7d1cd873543944caaa403a0238c90419 P 5f0ffac03be84e5bb3ca54739a6348b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:22:25.040673 18212 raft_consensus.cc:482] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:22:25.040786 18212 raft_consensus.cc:504] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0ffac03be84e5bb3ca54739a6348b3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 36777 } } peers { permanent_uuid: "ce88ae5c318847308daeea01af29316d" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 35521 } } peers { permanent_uuid: "ae8d062d1fa847079b104eb3b54b659e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 38811 } }
I20260924 15:22:25.041136 18212 leader_election.cc:288] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f0ffac03be84e5bb3ca54739a6348b3 (127.31.12.65:36777), ce88ae5c318847308daeea01af29316d (127.31.12.67:35521)
I20260924 15:22:25.045049 18203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.041391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:42033 (local address 127.31.12.65:36777)
0924 15:22:25.041460 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:25.041462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:25.041473 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:22:25.041559 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:25.041561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:25.041572 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:25.041663 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:25.041842 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.042552 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.044082 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.044655 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.044731 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.044739 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:25.044940 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 15:22:25.044978 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:22:25.045049 18202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.041409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:33125 (local address 127.31.12.67:35521)
0924 15:22:25.041482 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:25.041484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:25.041648 (+ 164us) server_negotiation.cc:407] Connection header received
0924 15:22:25.041703 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:25.041704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:25.041712 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:25.041758 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:25.041915 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.042546 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.044086 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.044655 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.044731 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.044739 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:25.044942 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 15:22:25.044982 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 15:22:25.045058 18057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.041342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:36777 (local address 127.31.12.66:42033)
0924 15:22:25.041409 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:25.041421 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:25.041493 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:25.041681 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:25.041682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:25.041684 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:25.041776 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:25.041778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.042629 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.042630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:25.043987 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:25.043990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.044737 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.044738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:25.044819 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.044822 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:22:25.044889 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 15:22:25.044949 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":49}
I20260924 15:22:25.045063 18214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.041358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:35521 (local address 127.31.12.66:33125)
0924 15:22:25.041545 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:25.041561 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:25.041634 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:25.041777 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:25.041778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:25.041780 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:25.041836 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:25.041837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.042629 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.042630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:25.043987 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:25.043990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.044738 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.044738 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:25.044818 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.044823 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:22:25.044889 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 15:22:25.044952 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":102,"thread_start_us":140,"threads_started":1}
I20260924 15:22:25.045508 17872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1cd873543944caaa403a0238c90419" candidate_uuid: "ae8d062d1fa847079b104eb3b54b659e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f0ffac03be84e5bb3ca54739a6348b3" is_pre_election: true
I20260924 15:22:25.045692 17872 raft_consensus.cc:2454] T 7d1cd873543944caaa403a0238c90419 P 5f0ffac03be84e5bb3ca54739a6348b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8d062d1fa847079b104eb3b54b659e in term 0.
I20260924 15:22:25.045775 18145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1cd873543944caaa403a0238c90419" candidate_uuid: "ae8d062d1fa847079b104eb3b54b659e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce88ae5c318847308daeea01af29316d" is_pre_election: true
I20260924 15:22:25.045946 18145 raft_consensus.cc:2454] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae8d062d1fa847079b104eb3b54b659e in term 0.
I20260924 15:22:25.045946 17941 leader_election.cc:302] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f0ffac03be84e5bb3ca54739a6348b3, ae8d062d1fa847079b104eb3b54b659e; no voters:
I20260924 15:22:25.046098 18212 raft_consensus.cc:2781] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:22:25.046185 18212 raft_consensus.cc:482] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:22:25.046208 18212 raft_consensus.cc:3037] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:25.046887 18212 raft_consensus.cc:504] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0ffac03be84e5bb3ca54739a6348b3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 36777 } } peers { permanent_uuid: "ce88ae5c318847308daeea01af29316d" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 35521 } } peers { permanent_uuid: "ae8d062d1fa847079b104eb3b54b659e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 38811 } }
I20260924 15:22:25.047029 18212 leader_election.cc:288] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [CANDIDATE]: Term 1 election: Requested vote from peers 5f0ffac03be84e5bb3ca54739a6348b3 (127.31.12.65:36777), ce88ae5c318847308daeea01af29316d (127.31.12.67:35521)
W20260924 15:22:25.047040 18192 tablet.cc:2369] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:22:25.047194 17872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1cd873543944caaa403a0238c90419" candidate_uuid: "ae8d062d1fa847079b104eb3b54b659e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f0ffac03be84e5bb3ca54739a6348b3"
I20260924 15:22:25.047277 17872 raft_consensus.cc:3037] T 7d1cd873543944caaa403a0238c90419 P 5f0ffac03be84e5bb3ca54739a6348b3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:25.047359 18145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d1cd873543944caaa403a0238c90419" candidate_uuid: "ae8d062d1fa847079b104eb3b54b659e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce88ae5c318847308daeea01af29316d"
I20260924 15:22:25.047458 18145 raft_consensus.cc:3037] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:25.048046 17872 raft_consensus.cc:2454] T 7d1cd873543944caaa403a0238c90419 P 5f0ffac03be84e5bb3ca54739a6348b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8d062d1fa847079b104eb3b54b659e in term 1.
I20260924 15:22:25.048137 18145 raft_consensus.cc:2454] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae8d062d1fa847079b104eb3b54b659e in term 1.
I20260924 15:22:25.048198 17941 leader_election.cc:302] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f0ffac03be84e5bb3ca54739a6348b3, ae8d062d1fa847079b104eb3b54b659e; no voters:
I20260924 15:22:25.048285 18212 raft_consensus.cc:2781] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:25.048429 18212 raft_consensus.cc:686] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [term 1 LEADER]: Becoming Leader. State: Replica: ae8d062d1fa847079b104eb3b54b659e, State: Running, Role: LEADER
I20260924 15:22:25.048518 18212 consensus_queue.cc:237] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0ffac03be84e5bb3ca54739a6348b3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 36777 } } peers { permanent_uuid: "ce88ae5c318847308daeea01af29316d" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 35521 } } peers { permanent_uuid: "ae8d062d1fa847079b104eb3b54b659e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 38811 } }
I20260924 15:22:25.049510 17721 catalog_manager.cc:5495] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e reported cstate change: term changed from 0 to 1, leader changed from <none> to ae8d062d1fa847079b104eb3b54b659e (127.31.12.66). New cstate: current_term: 1 leader_uuid: "ae8d062d1fa847079b104eb3b54b659e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f0ffac03be84e5bb3ca54739a6348b3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 36777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce88ae5c318847308daeea01af29316d" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 35521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8d062d1fa847079b104eb3b54b659e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 38811 } health_report { overall_health: HEALTHY } } }
I20260924 15:22:25.109266 18204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.106930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38906 (local address 127.31.12.66:38811)
0924 15:22:25.106981 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:25.106983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:25.106992 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:22:25.107076 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:25.107077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:25.107133 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:25.107203 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:22:25.107377 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.107835 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.108575 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.108763 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.108817 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.108926 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:22:25.109011 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:22:25.109074 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:25.109113 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 15:22:25.109148 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 15:22:25.111145 17872 raft_consensus.cc:1261] T 7d1cd873543944caaa403a0238c90419 P 5f0ffac03be84e5bb3ca54739a6348b3 [term 1 FOLLOWER]: Refusing update from remote peer ae8d062d1fa847079b104eb3b54b659e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:22:25.111146 18145 raft_consensus.cc:1261] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d [term 1 FOLLOWER]: Refusing update from remote peer ae8d062d1fa847079b104eb3b54b659e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:22:25.111516 18212 consensus_queue.cc:1035] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f0ffac03be84e5bb3ca54739a6348b3" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 36777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:22:25.111680 18215 consensus_queue.cc:1035] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce88ae5c318847308daeea01af29316d" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 35521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:22:25.114418 18219 mvcc.cc:205] Tried to move back new op lower bound from 7332918661572345856 to 7332918661319307264. Current Snapshot: MvccSnapshot[applied={T|T < 7332918661572345856}]
I20260924 15:22:25.116787 17721 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:22:25.116879 17721 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:22:25.117660 17721 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25b68c90e7bf44dcbf5d93553c1b63db: Sending DeleteTablet for 3 replicas of tablet 7d1cd873543944caaa403a0238c90419
I20260924 15:22:25.117996 17851 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d1cd873543944caaa403a0238c90419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:22:25 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:54350
I20260924 15:22:25.117998 17988 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d1cd873543944caaa403a0238c90419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:22:25 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:38900
I20260924 15:22:25.118397 18124 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d1cd873543944caaa403a0238c90419 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:22:25 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:46250
I20260924 15:22:25.118464 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 17787
W20260924 15:22:25.124415 17707 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.12.65:36777 (error 108)
I20260924 15:22:25.124459 18228 tablet_replica.cc:330] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e: stopping tablet replica
I20260924 15:22:25.124560 18228 raft_consensus.cc:2229] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:22:25.124666 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 17924
I20260924 15:22:25.124686 18228 raft_consensus.cc:2258] T 7d1cd873543944caaa403a0238c90419 P ae8d062d1fa847079b104eb3b54b659e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:22:25.125289 18230 tablet_replica.cc:330] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d: stopping tablet replica
I20260924 15:22:25.125391 18230 raft_consensus.cc:2229] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:22:25.125511 18230 raft_consensus.cc:2258] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:22:25.125784 18199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.125667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:36777 (local address 127.0.0.1:54364)
0924 15:22:25.125713 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:25.125745 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.65:36777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 15:22:25.125907 17707 catalog_manager.cc:4584] TS 5f0ffac03be84e5bb3ca54739a6348b3 (127.31.12.65:36777): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7d1cd873543944caaa403a0238c90419: Network error: Client connection negotiation failed: client connection to 127.31.12.65:36777: connect: Connection refused (error 111)
I20260924 15:22:25.126145 18230 ts_tablet_manager.cc:1905] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:22:25.127717 18230 ts_tablet_manager.cc:1918] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:22:25.127789 18230 log.cc:1196] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/7d1cd873543944caaa403a0238c90419
I20260924 15:22:25.128067 18230 ts_tablet_manager.cc:1939] T 7d1cd873543944caaa403a0238c90419 P ce88ae5c318847308daeea01af29316d: Deleting consensus metadata
I20260924 15:22:25.128485 17708 catalog_manager.cc:4841] TS ce88ae5c318847308daeea01af29316d (127.31.12.67:35521): tablet 7d1cd873543944caaa403a0238c90419 (table test-table [id=6373cc8fc1b846c4af2ce7bff493c0ed]) successfully deleted
I20260924 15:22:25.130501 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 18060
I20260924 15:22:25.130515 18199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.130401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:38811 (local address 127.0.0.1:38912)
0924 15:22:25.130458 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:25.130491 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.66:38811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 15:22:25.135000 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 17690
2026-09-24T15:22:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2124 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T15:22:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:25Z Disabled control of system clock
I20260924 15:22:25.162724 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:33483
--webserver_interface=127.31.12.126
--webserver_port=0
--builtin_ntp_servers=127.31.12.84:33443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:33483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:22:25.243106 18235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:25.243304 18235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:25.243325 18235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:25.244410 18235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:25.244442 18235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:25.244455 18235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:25.244467 18235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:25.244479 18235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:25.245784 18235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:33483
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:33483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:25.245976 18235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:25.246187 18235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:25.250802 18243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:25.250829 18241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:25.250869 18235 server_base.cc:1034] running on GCE node
W20260924 15:22:25.250818 18240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:25.251259 18235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:25.251483 18235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:25.252628 18235 hybrid_clock.cc:648] HybridClock initialized: now 1790263345252617 us; error 27 us; skew 500 ppm
I20260924 15:22:25.259224 18235 webserver.cc:460] Webserver started at http://127.31.12.126:37347/ using document root <none> and password file <none>
I20260924 15:22:25.259413 18235 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:25.259452 18235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:25.259539 18235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:25.260488 18235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "4bdad8dbee2245bbb85b63180fc1103f"
format_stamp: "Formatted at 2026-09-24 15:22:25 on dist-test-slave-zwbb"
server_key: "fa99a979cd023d2d7184a2e48adf6af3"
server_key_iv: "19ad79c4f00b7a871c7f3336e9c5429c"
server_key_version: "encryptionkey@0"
I20260924 15:22:25.260766 18235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "4bdad8dbee2245bbb85b63180fc1103f"
format_stamp: "Formatted at 2026-09-24 15:22:25 on dist-test-slave-zwbb"
server_key: "fa99a979cd023d2d7184a2e48adf6af3"
server_key_iv: "19ad79c4f00b7a871c7f3336e9c5429c"
server_key_version: "encryptionkey@0"
I20260924 15:22:25.262022 18235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:22:25.262672 18249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:25.262862 18235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:25.262938 18235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "4bdad8dbee2245bbb85b63180fc1103f"
format_stamp: "Formatted at 2026-09-24 15:22:25 on dist-test-slave-zwbb"
server_key: "fa99a979cd023d2d7184a2e48adf6af3"
server_key_iv: "19ad79c4f00b7a871c7f3336e9c5429c"
server_key_version: "encryptionkey@0"
I20260924 15:22:25.262992 18235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:25.276363 18235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:25.276633 18235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:25.276731 18235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:25.280462 18235 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:33483
I20260924 15:22:25.280489 18302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:33483 every 8 connection(s)
I20260924 15:22:25.280808 18235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:25.281766 18304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:25.283903 18304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f: Bootstrap starting.
I20260924 15:22:25.284469 18304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:25.284765 18304 log.cc:824] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:25.285535 18304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f: No bootstrap required, opened a new log
I20260924 15:22:25.286760 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 18235
I20260924 15:22:25.286846 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:25.287055 31793 external_mini_cluster.cc:1428] Setting key d0b38353e72817075bae88cea0f540d9
I20260924 15:22:25.287238 18304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bdad8dbee2245bbb85b63180fc1103f" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33483 } }
I20260924 15:22:25.287396 18304 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:25.287431 18304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bdad8dbee2245bbb85b63180fc1103f, State: Initialized, Role: FOLLOWER
I20260924 15:22:25.287799 18304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bdad8dbee2245bbb85b63180fc1103f" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33483 } }
I20260924 15:22:25.287905 18304 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:25.287940 18304 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:25.287981 18304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:25.288985 18304 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bdad8dbee2245bbb85b63180fc1103f" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33483 } }
I20260924 15:22:25.289094 18304 leader_election.cc:302] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bdad8dbee2245bbb85b63180fc1103f; no voters:
I20260924 15:22:25.289242 18304 leader_election.cc:288] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:25.289331 18309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:25.289529 18309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f [term 1 LEADER]: Becoming Leader. State: Replica: 4bdad8dbee2245bbb85b63180fc1103f, State: Running, Role: LEADER
I20260924 15:22:25.289515 18304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:25.289691 18309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bdad8dbee2245bbb85b63180fc1103f" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33483 } }
I20260924 15:22:25.290712 18311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bdad8dbee2245bbb85b63180fc1103f. Latest consensus state: current_term: 1 leader_uuid: "4bdad8dbee2245bbb85b63180fc1103f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bdad8dbee2245bbb85b63180fc1103f" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33483 } } }
I20260924 15:22:25.290795 18311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:25.291218 18310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bdad8dbee2245bbb85b63180fc1103f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bdad8dbee2245bbb85b63180fc1103f" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33483 } } }
I20260924 15:22:25.291065 18323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:25.291338 18310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:25.291811 18323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:25.293073 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.287531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50136 (local address 127.31.12.126:33483)
0924 15:22:25.287731 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:25.287740 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:25.287772 (+ 32us) server_negotiation.cc:407] Connection header received
0924 15:22:25.287887 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:25.287895 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:25.287974 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:25.288125 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:25.288304 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.289762 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.290950 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.291561 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.291695 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.292309 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:25.292312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:25.292313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:25.292521 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:25.292562 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:25.292576 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:25.292642 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:25.292759 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 15:22:25.292836 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
I20260924 15:22:25.293809 18323 catalog_manager.cc:1347] Generated new cluster ID: 9c85153dbfde45598f50a8d787a24aca
I20260924 15:22:25.293893 18323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:25.305496 18323 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:25.306094 18323 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:25.312670 18323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f: Generated new TSK 0
I20260924 15:22:25.312824 18323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:25.315495 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:33483
--builtin_ntp_servers=127.31.12.84:33443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:22:25.394873 18328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:25.395094 18328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:25.395126 18328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:25.396278 18328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:25.396341 18328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:25.396384 18328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:22:25.397778 18328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:33483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:25.398048 18328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:25.398331 18328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:25.403786 18334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:25.403786 18336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:25.403985 18328 server_base.cc:1034] running on GCE node
W20260924 15:22:25.404171 18333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:25.404414 18328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:25.404659 18328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:25.405814 18328 hybrid_clock.cc:648] HybridClock initialized: now 1790263345405783 us; error 46 us; skew 500 ppm
I20260924 15:22:25.407063 18328 webserver.cc:460] Webserver started at http://127.31.12.65:36913/ using document root <none> and password file <none>
I20260924 15:22:25.407323 18328 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:25.407405 18328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:25.407569 18328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:25.408605 18328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "476a6a56a68c4d1f82f825d8e3858063"
format_stamp: "Formatted at 2026-09-24 15:22:25 on dist-test-slave-zwbb"
server_key: "f06386e21c3bc70984b7f56ebdb20361"
server_key_iv: "a11256e5718b3ee5d4b50119372f7a58"
server_key_version: "encryptionkey@0"
I20260924 15:22:25.408974 18328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "476a6a56a68c4d1f82f825d8e3858063"
format_stamp: "Formatted at 2026-09-24 15:22:25 on dist-test-slave-zwbb"
server_key: "f06386e21c3bc70984b7f56ebdb20361"
server_key_iv: "a11256e5718b3ee5d4b50119372f7a58"
server_key_version: "encryptionkey@0"
I20260924 15:22:25.410427 18328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:22:25.411211 18342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:25.411388 18328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:22:25.411509 18328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "476a6a56a68c4d1f82f825d8e3858063"
format_stamp: "Formatted at 2026-09-24 15:22:25 on dist-test-slave-zwbb"
server_key: "f06386e21c3bc70984b7f56ebdb20361"
server_key_iv: "a11256e5718b3ee5d4b50119372f7a58"
server_key_version: "encryptionkey@0"
I20260924 15:22:25.411597 18328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:25.420749 18328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:25.421015 18328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:25.421161 18328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:25.421383 18328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:25.421746 18328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:22:25.421808 18328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:25.421845 18328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:22:25.421872 18328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:25.427515 18328 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:35329
I20260924 15:22:25.427591 18457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:35329 every 8 connection(s)
I20260924 15:22:25.427878 18328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:22:25.430460 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 18328
I20260924 15:22:25.430583 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:22:25.430765 31793 external_mini_cluster.cc:1428] Setting key da49acc83611ed23ae9ddf449798294b
I20260924 15:22:25.431622 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:33483
--builtin_ntp_servers=127.31.12.84:33443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 15:22:25.433092 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.428585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:39811 (local address 127.31.12.126:33483)
0924 15:22:25.428657 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:25.428658 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:25.428892 (+ 234us) server_negotiation.cc:407] Connection header received
0924 15:22:25.429421 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:25.429423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:25.429450 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:25.429511 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:25.430184 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.430744 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.431642 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.431923 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.431996 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.432753 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:25.432754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:25.432754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:25.432759 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:25.432775 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:25.432780 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:25.432833 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:25.432943 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 15:22:25.432973 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 15:22:25.433310 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.428570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33483 (local address 127.31.12.65:39811)
0924 15:22:25.428766 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:25.428796 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:25.429296 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:25.429583 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:25.429587 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:25.429597 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:25.430085 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:25.430090 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.430838 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.430840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:25.431552 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:25.431554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.432004 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.432005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:25.432129 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.432598 (+ 469us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:22:25.432598 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:25.432634 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:22:25.432660 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:25.432850 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:25.432857 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:25.432860 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:22:25.432927 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 15:22:25.433056 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260924 15:22:25.433812 18458 heartbeater.cc:347] Connected to a master server at 127.31.12.126:33483
I20260924 15:22:25.433918 18458 heartbeater.cc:464] Registering TS with master...
I20260924 15:22:25.434087 18458 heartbeater.cc:511] Master 127.31.12.126:33483 requested a full tablet report, sending...
I20260924 15:22:25.434630 18266 ts_manager.cc:194] Registered new tserver with Master: 476a6a56a68c4d1f82f825d8e3858063 (127.31.12.65:35329)
I20260924 15:22:25.435194 18266 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.12.65:39811
W20260924 15:22:25.511760 18461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:25.511998 18461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:25.512033 18461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:25.513177 18461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:25.513243 18461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:25.513284 18461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:22:25.514680 18461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:33483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:25.514941 18461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:25.515242 18461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:25.520269 18466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:25.520237 18467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:25.520247 18469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:25.520356 18461 server_base.cc:1034] running on GCE node
I20260924 15:22:25.520630 18461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:25.520884 18461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:25.522048 18461 hybrid_clock.cc:648] HybridClock initialized: now 1790263345522044 us; error 37 us; skew 500 ppm
I20260924 15:22:25.523306 18461 webserver.cc:460] Webserver started at http://127.31.12.66:41289/ using document root <none> and password file <none>
I20260924 15:22:25.523551 18461 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:25.523624 18461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:25.523779 18461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:25.524811 18461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "190a8eb1e7244fb0899a075100cd6d10"
format_stamp: "Formatted at 2026-09-24 15:22:25 on dist-test-slave-zwbb"
server_key: "8bc821f5ed7cb5d4e231fed9e069b201"
server_key_iv: "624021e2a475bafe6e2fa85fd7263ac3"
server_key_version: "encryptionkey@0"
I20260924 15:22:25.525209 18461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "190a8eb1e7244fb0899a075100cd6d10"
format_stamp: "Formatted at 2026-09-24 15:22:25 on dist-test-slave-zwbb"
server_key: "8bc821f5ed7cb5d4e231fed9e069b201"
server_key_iv: "624021e2a475bafe6e2fa85fd7263ac3"
server_key_version: "encryptionkey@0"
I20260924 15:22:25.526553 18461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:22:25.527281 18475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:25.527446 18461 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:25.527527 18461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "190a8eb1e7244fb0899a075100cd6d10"
format_stamp: "Formatted at 2026-09-24 15:22:25 on dist-test-slave-zwbb"
server_key: "8bc821f5ed7cb5d4e231fed9e069b201"
server_key_iv: "624021e2a475bafe6e2fa85fd7263ac3"
server_key_version: "encryptionkey@0"
I20260924 15:22:25.527580 18461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:25.538242 18461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:25.538461 18461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:25.538561 18461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:25.538743 18461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:25.539001 18461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:22:25.539047 18461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:25.539098 18461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:22:25.539122 18461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:25.544648 18461 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:42231
I20260924 15:22:25.544716 18589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:42231 every 8 connection(s)
I20260924 15:22:25.545010 18461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:22:25.547444 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 18461
I20260924 15:22:25.547550 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:22:25.547747 31793 external_mini_cluster.cc:1428] Setting key a1e20bdfc7569ffec81bd4f3ca43982b
I20260924 15:22:25.548768 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:33483
--builtin_ntp_servers=127.31.12.84:33443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 15:22:25.550624 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.545739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:45159 (local address 127.31.12.126:33483)
0924 15:22:25.545800 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:25.545802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:25.546032 (+ 230us) server_negotiation.cc:407] Connection header received
0924 15:22:25.546461 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:25.546462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:25.546471 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:25.546526 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:25.547149 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.547938 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.549155 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.549505 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.549573 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.550402 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:25.550403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:25.550403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:25.550407 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:25.550421 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:25.550425 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:25.550467 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:25.550549 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 15:22:25.550570 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 15:22:25.550880 18592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.545727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33483 (local address 127.31.12.66:45159)
0924 15:22:25.545912 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:25.545950 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:25.546383 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:25.546574 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:25.546576 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:25.546584 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:25.547041 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:25.547045 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.548128 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.548129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:25.549078 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:25.549081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.549580 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.549582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:25.549692 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.550227 (+ 535us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:22:25.550228 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:25.550269 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:22:25.550302 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:25.550469 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:25.550474 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:25.550477 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:22:25.550542 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 15:22:25.550646 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260924 15:22:25.551277 18590 heartbeater.cc:347] Connected to a master server at 127.31.12.126:33483
I20260924 15:22:25.551402 18590 heartbeater.cc:464] Registering TS with master...
I20260924 15:22:25.551597 18590 heartbeater.cc:511] Master 127.31.12.126:33483 requested a full tablet report, sending...
I20260924 15:22:25.551971 18266 ts_manager.cc:194] Registered new tserver with Master: 190a8eb1e7244fb0899a075100cd6d10 (127.31.12.66:42231)
I20260924 15:22:25.552359 18266 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.12.66:45159
W20260924 15:22:25.631285 18593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:25.631510 18593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:25.631543 18593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:25.632616 18593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:25.632676 18593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:25.632715 18593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:22:25.634086 18593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:33483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:25.634348 18593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:25.634627 18593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:25.639528 18599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:25.639532 18601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:25.639743 18593 server_base.cc:1034] running on GCE node
W20260924 15:22:25.639577 18598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:25.639986 18593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:25.640235 18593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:25.641404 18593 hybrid_clock.cc:648] HybridClock initialized: now 1790263345641387 us; error 28 us; skew 500 ppm
I20260924 15:22:25.642607 18593 webserver.cc:460] Webserver started at http://127.31.12.67:44107/ using document root <none> and password file <none>
I20260924 15:22:25.642879 18593 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:25.642951 18593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:25.643100 18593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:25.644114 18593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "9b3cd5fcc0294fe4a8acae8034f80c56"
format_stamp: "Formatted at 2026-09-24 15:22:25 on dist-test-slave-zwbb"
server_key: "9fab2a630c70bdcffe82fdcb71056548"
server_key_iv: "16091e5fd483221e6014d894d6b19a63"
server_key_version: "encryptionkey@0"
I20260924 15:22:25.644471 18593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "9b3cd5fcc0294fe4a8acae8034f80c56"
format_stamp: "Formatted at 2026-09-24 15:22:25 on dist-test-slave-zwbb"
server_key: "9fab2a630c70bdcffe82fdcb71056548"
server_key_iv: "16091e5fd483221e6014d894d6b19a63"
server_key_version: "encryptionkey@0"
I20260924 15:22:25.645877 18593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:22:25.646581 18607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:25.646749 18593 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:25.646836 18593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "9b3cd5fcc0294fe4a8acae8034f80c56"
format_stamp: "Formatted at 2026-09-24 15:22:25 on dist-test-slave-zwbb"
server_key: "9fab2a630c70bdcffe82fdcb71056548"
server_key_iv: "16091e5fd483221e6014d894d6b19a63"
server_key_version: "encryptionkey@0"
I20260924 15:22:25.646932 18593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:25.667819 18593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:25.668121 18593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:25.668256 18593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:25.668479 18593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:25.668779 18593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:22:25.668859 18593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:25.668889 18593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:22:25.668920 18593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:25.674590 18593 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:46101
I20260924 15:22:25.674664 18721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:46101 every 8 connection(s)
I20260924 15:22:25.674973 18593 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:22:25.680042 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.675716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:47265 (local address 127.31.12.126:33483)
0924 15:22:25.675807 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:25.675809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:25.676008 (+ 199us) server_negotiation.cc:407] Connection header received
0924 15:22:25.676466 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:25.676467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:25.676477 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:25.676530 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:25.677171 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.677738 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.678673 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.678861 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.678915 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.679845 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:25.679846 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:25.679846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:25.679850 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:25.679861 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:25.679865 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:25.679904 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:25.679966 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 15:22:25.679986 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 15:22:25.680231 18724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.675667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:33483 (local address 127.31.12.67:47265)
0924 15:22:25.675885 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:25.675930 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:25.676369 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:25.676594 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:25.676597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:25.676605 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:25.677103 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:25.677107 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.677820 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.677821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:25.678595 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:25.678598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.678948 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.678949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:25.679055 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.679728 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:22:25.679728 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:25.679755 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:22:25.679776 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:25.679906 (+ 130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:25.679909 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:25.679911 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:22:25.679955 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 15:22:25.680024 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":110,"threads_started":1}
I20260924 15:22:25.680790 18722 heartbeater.cc:347] Connected to a master server at 127.31.12.126:33483
I20260924 15:22:25.680902 18722 heartbeater.cc:464] Registering TS with master...
I20260924 15:22:25.681085 18722 heartbeater.cc:511] Master 127.31.12.126:33483 requested a full tablet report, sending...
I20260924 15:22:25.681542 18266 ts_manager.cc:194] Registered new tserver with Master: 9b3cd5fcc0294fe4a8acae8034f80c56 (127.31.12.67:46101)
I20260924 15:22:25.681902 18266 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.12.67:47265
I20260924 15:22:25.684283 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 18593
I20260924 15:22:25.684393 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:22:25.684587 31793 external_mini_cluster.cc:1428] Setting key b5810049265a97e5d4a8d7e15b2f4f62
I20260924 15:22:25.685585 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:22:25.689694 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.686909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50142 (local address 127.31.12.126:33483)
0924 15:22:25.686981 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:25.686983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:25.687091 (+ 108us) server_negotiation.cc:407] Connection header received
0924 15:22:25.687125 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:25.687126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:25.687137 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:25.687181 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:25.687313 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.687764 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.688479 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.688649 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.688701 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.689367 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:25.689368 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:25.689368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:25.689372 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:25.689382 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:25.689386 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:25.689510 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:25.689556 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 15:22:25.689565 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:22:25.691437 18266 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:22:25.692859 18266 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:22:25.698832 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.695263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45178 (local address 127.31.12.67:46101)
0924 15:22:25.695411 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:25.695415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:25.695426 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:22:25.695463 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:25.695465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:25.695480 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:25.695731 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:25.695776 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.696782 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.697846 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.698454 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.698553 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.698638 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:25.698723 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 15:22:25.698756 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260924 15:22:25.698824 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.694962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:46101 (local address 127.0.0.1:45178)
0924 15:22:25.695184 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:25.695195 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:25.695244 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:25.695586 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:25.695593 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:25.695594 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:25.695656 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:25.695658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.696874 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.696875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:25.697746 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:25.697749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.698553 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.698554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:25.698632 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.698646 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 15:22:25.698712 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 15:22:25.698760 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":113,"threads_started":1}
I20260924 15:22:25.699167 18731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.694934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:35329 (local address 127.0.0.1:40038)
0924 15:22:25.695125 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:25.695164 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:25.695227 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:25.695734 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:25.695735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:25.695736 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:25.695769 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:25.695770 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.697018 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.697019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:25.698083 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:25.698085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.698946 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.698947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:25.699011 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.699014 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:22:25.699045 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 15:22:25.699090 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":63,"threads_started":1}
I20260924 15:22:25.699167 18733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.695134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:42231 (local address 127.0.0.1:53580)
0924 15:22:25.695411 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:25.695421 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:25.695477 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:25.695645 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:25.695646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:25.695648 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:25.695692 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:25.695694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.696715 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.696717 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:25.697929 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:25.697932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.698952 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.698953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:25.699011 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.699016 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:22:25.699071 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 15:22:25.699101 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":74,"threads_started":1}
I20260924 15:22:25.699247 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.695237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53580 (local address 127.31.12.66:42231)
0924 15:22:25.695393 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:25.695397 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:25.695487 (+ 90us) server_negotiation.cc:407] Connection header received
0924 15:22:25.695539 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:25.695540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:25.695552 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:25.695647 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:25.695771 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.696632 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.698021 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.698765 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.698834 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.698924 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:25.699107 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 15:22:25.699152 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260924 15:22:25.699615 18732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.695016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40038 (local address 127.31.12.65:35329)
0924 15:22:25.695601 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:25.695605 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:25.695614 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:22:25.695649 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:25.695651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:25.695664 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:25.695736 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:25.695832 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.696942 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.698165 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.698841 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.699375 (+ 534us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.699473 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:25.699498 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 15:22:25.699538 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":80,"threads_started":1}
I20260924 15:22:25.699747 18655 tablet_service.cc:1467] Processing CreateTablet for tablet bc4adb480a0c47ecabd1af4b535c3326 (DEFAULT_TABLE table=test-table [id=5527b9bcc07f49289097ae1a169cf4ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:22:25.699883 18523 tablet_service.cc:1467] Processing CreateTablet for tablet bc4adb480a0c47ecabd1af4b535c3326 (DEFAULT_TABLE table=test-table [id=5527b9bcc07f49289097ae1a169cf4ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:22:25.700115 18655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc4adb480a0c47ecabd1af4b535c3326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:25.700208 18523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc4adb480a0c47ecabd1af4b535c3326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:25.700206 18390 tablet_service.cc:1467] Processing CreateTablet for tablet bc4adb480a0c47ecabd1af4b535c3326 (DEFAULT_TABLE table=test-table [id=5527b9bcc07f49289097ae1a169cf4ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:22:25.700542 18390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc4adb480a0c47ecabd1af4b535c3326. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:25.703287 18736 tablet_bootstrap.cc:492] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063: Bootstrap starting.
I20260924 15:22:25.703806 18737 tablet_bootstrap.cc:492] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56: Bootstrap starting.
I20260924 15:22:25.703902 18736 tablet_bootstrap.cc:654] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:25.704200 18736 log.cc:824] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:25.704435 18737 tablet_bootstrap.cc:654] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:25.704814 18737 log.cc:824] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:25.705013 18736 tablet_bootstrap.cc:492] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063: No bootstrap required, opened a new log
I20260924 15:22:25.705091 18736 ts_tablet_manager.cc:1397] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:22:25.705142 18739 tablet_bootstrap.cc:492] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10: Bootstrap starting.
I20260924 15:22:25.705550 18737 tablet_bootstrap.cc:492] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56: No bootstrap required, opened a new log
I20260924 15:22:25.705616 18737 ts_tablet_manager.cc:1397] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:22:25.705803 18739 tablet_bootstrap.cc:654] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:25.706104 18739 log.cc:824] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:25.706940 18739 tablet_bootstrap.cc:492] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10: No bootstrap required, opened a new log
I20260924 15:22:25.707010 18739 ts_tablet_manager.cc:1397] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 15:22:25.707469 18736 raft_consensus.cc:348] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a6a56a68c4d1f82f825d8e3858063" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 35329 } } peers { permanent_uuid: "9b3cd5fcc0294fe4a8acae8034f80c56" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 46101 } } peers { permanent_uuid: "190a8eb1e7244fb0899a075100cd6d10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 42231 } }
I20260924 15:22:25.707612 18736 raft_consensus.cc:374] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:25.707665 18736 raft_consensus.cc:729] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476a6a56a68c4d1f82f825d8e3858063, State: Initialized, Role: FOLLOWER
I20260924 15:22:25.707768 18736 consensus_queue.cc:260] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a6a56a68c4d1f82f825d8e3858063" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 35329 } } peers { permanent_uuid: "9b3cd5fcc0294fe4a8acae8034f80c56" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 46101 } } peers { permanent_uuid: "190a8eb1e7244fb0899a075100cd6d10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 42231 } }
I20260924 15:22:25.708016 18736 ts_tablet_manager.cc:1428] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 15:22:25.708137 18458 heartbeater.cc:503] Master 127.31.12.126:33483 was elected leader, sending a full tablet report...
I20260924 15:22:25.708231 18737 raft_consensus.cc:348] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a6a56a68c4d1f82f825d8e3858063" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 35329 } } peers { permanent_uuid: "9b3cd5fcc0294fe4a8acae8034f80c56" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 46101 } } peers { permanent_uuid: "190a8eb1e7244fb0899a075100cd6d10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 42231 } }
I20260924 15:22:25.708385 18737 raft_consensus.cc:374] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:25.708418 18737 raft_consensus.cc:729] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b3cd5fcc0294fe4a8acae8034f80c56, State: Initialized, Role: FOLLOWER
I20260924 15:22:25.708508 18737 consensus_queue.cc:260] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a6a56a68c4d1f82f825d8e3858063" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 35329 } } peers { permanent_uuid: "9b3cd5fcc0294fe4a8acae8034f80c56" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 46101 } } peers { permanent_uuid: "190a8eb1e7244fb0899a075100cd6d10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 42231 } }
I20260924 15:22:25.708732 18737 ts_tablet_manager.cc:1428] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 15:22:25.708760 18722 heartbeater.cc:503] Master 127.31.12.126:33483 was elected leader, sending a full tablet report...
I20260924 15:22:25.709465 18739 raft_consensus.cc:348] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a6a56a68c4d1f82f825d8e3858063" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 35329 } } peers { permanent_uuid: "9b3cd5fcc0294fe4a8acae8034f80c56" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 46101 } } peers { permanent_uuid: "190a8eb1e7244fb0899a075100cd6d10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 42231 } }
I20260924 15:22:25.709636 18739 raft_consensus.cc:374] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:25.709676 18739 raft_consensus.cc:729] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 190a8eb1e7244fb0899a075100cd6d10, State: Initialized, Role: FOLLOWER
I20260924 15:22:25.709782 18739 consensus_queue.cc:260] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a6a56a68c4d1f82f825d8e3858063" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 35329 } } peers { permanent_uuid: "9b3cd5fcc0294fe4a8acae8034f80c56" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 46101 } } peers { permanent_uuid: "190a8eb1e7244fb0899a075100cd6d10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 42231 } }
I20260924 15:22:25.709966 18739 ts_tablet_manager.cc:1428] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 15:22:25.710048 18590 heartbeater.cc:503] Master 127.31.12.126:33483 was elected leader, sending a full tablet report...
W20260924 15:22:25.795568 18591 tablet.cc:2369] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:22:25.842818 18743 raft_consensus.cc:482] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:22:25.842964 18743 raft_consensus.cc:504] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a6a56a68c4d1f82f825d8e3858063" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 35329 } } peers { permanent_uuid: "9b3cd5fcc0294fe4a8acae8034f80c56" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 46101 } } peers { permanent_uuid: "190a8eb1e7244fb0899a075100cd6d10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 42231 } }
I20260924 15:22:25.843359 18743 leader_election.cc:288] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b3cd5fcc0294fe4a8acae8034f80c56 (127.31.12.67:46101), 190a8eb1e7244fb0899a075100cd6d10 (127.31.12.66:42231)
I20260924 15:22:25.846395 18747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.843561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:46101 (local address 127.31.12.65:57911)
0924 15:22:25.843770 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:25.843782 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:25.843841 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:25.844032 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:25.844033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:25.844035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:25.844105 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:25.844106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.844763 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.844764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:25.845577 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:25.845579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.846124 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.846126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:25.846201 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.846205 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:22:25.846256 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 15:22:25.846287 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":17,"thread_start_us":127,"threads_started":1}
I20260924 15:22:25.846390 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.843635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:57911 (local address 127.31.12.67:46101)
0924 15:22:25.843685 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:25.843687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:25.843895 (+ 208us) server_negotiation.cc:407] Connection header received
0924 15:22:25.843929 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:25.843930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:25.843941 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:25.843999 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:25.844195 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.844645 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.845664 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.846047 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.846102 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.846109 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:25.846277 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 15:22:25.846322 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 15:22:25.846472 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.843637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:54075 (local address 127.31.12.66:42231)
0924 15:22:25.843702 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:25.843704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:25.843714 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:22:25.843749 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:25.843750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:25.843761 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:25.843838 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:25.844057 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.844619 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.845733 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.846108 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.846179 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.846187 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:25.846351 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 15:22:25.846396 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 15:22:25.846500 18460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.843561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:42231 (local address 127.31.12.65:54075)
0924 15:22:25.843628 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:25.843641 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:25.843688 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:25.843883 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:25.843885 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:25.843887 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:25.843978 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:25.843980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.844710 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.844711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:25.845629 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:25.845632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.846198 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.846199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:25.846270 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.846273 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:22:25.846328 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 15:22:25.846369 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":57}
I20260924 15:22:25.846729 18675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4adb480a0c47ecabd1af4b535c3326" candidate_uuid: "476a6a56a68c4d1f82f825d8e3858063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b3cd5fcc0294fe4a8acae8034f80c56" is_pre_election: true
I20260924 15:22:25.846807 18544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4adb480a0c47ecabd1af4b535c3326" candidate_uuid: "476a6a56a68c4d1f82f825d8e3858063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "190a8eb1e7244fb0899a075100cd6d10" is_pre_election: true
I20260924 15:22:25.846892 18675 raft_consensus.cc:2454] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 476a6a56a68c4d1f82f825d8e3858063 in term 0.
I20260924 15:22:25.846935 18544 raft_consensus.cc:2454] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 476a6a56a68c4d1f82f825d8e3858063 in term 0.
I20260924 15:22:25.847134 18346 leader_election.cc:302] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 476a6a56a68c4d1f82f825d8e3858063, 9b3cd5fcc0294fe4a8acae8034f80c56; no voters:
I20260924 15:22:25.847249 18743 raft_consensus.cc:2781] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:22:25.847302 18743 raft_consensus.cc:482] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:22:25.847338 18743 raft_consensus.cc:3037] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:25.848028 18743 raft_consensus.cc:504] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a6a56a68c4d1f82f825d8e3858063" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 35329 } } peers { permanent_uuid: "9b3cd5fcc0294fe4a8acae8034f80c56" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 46101 } } peers { permanent_uuid: "190a8eb1e7244fb0899a075100cd6d10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 42231 } }
I20260924 15:22:25.848166 18743 leader_election.cc:288] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [CANDIDATE]: Term 1 election: Requested vote from peers 9b3cd5fcc0294fe4a8acae8034f80c56 (127.31.12.67:46101), 190a8eb1e7244fb0899a075100cd6d10 (127.31.12.66:42231)
I20260924 15:22:25.848342 18544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4adb480a0c47ecabd1af4b535c3326" candidate_uuid: "476a6a56a68c4d1f82f825d8e3858063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "190a8eb1e7244fb0899a075100cd6d10"
I20260924 15:22:25.848341 18675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc4adb480a0c47ecabd1af4b535c3326" candidate_uuid: "476a6a56a68c4d1f82f825d8e3858063" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b3cd5fcc0294fe4a8acae8034f80c56"
I20260924 15:22:25.848419 18544 raft_consensus.cc:3037] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:25.848419 18675 raft_consensus.cc:3037] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:25.849313 18544 raft_consensus.cc:2454] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 476a6a56a68c4d1f82f825d8e3858063 in term 1.
I20260924 15:22:25.849313 18675 raft_consensus.cc:2454] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 476a6a56a68c4d1f82f825d8e3858063 in term 1.
I20260924 15:22:25.849550 18346 leader_election.cc:302] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 476a6a56a68c4d1f82f825d8e3858063, 9b3cd5fcc0294fe4a8acae8034f80c56; no voters:
I20260924 15:22:25.849635 18743 raft_consensus.cc:2781] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:25.849774 18743 raft_consensus.cc:686] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [term 1 LEADER]: Becoming Leader. State: Replica: 476a6a56a68c4d1f82f825d8e3858063, State: Running, Role: LEADER
I20260924 15:22:25.849879 18743 consensus_queue.cc:237] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a6a56a68c4d1f82f825d8e3858063" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 35329 } } peers { permanent_uuid: "9b3cd5fcc0294fe4a8acae8034f80c56" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 46101 } } peers { permanent_uuid: "190a8eb1e7244fb0899a075100cd6d10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 42231 } }
I20260924 15:22:25.850764 18265 catalog_manager.cc:5495] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 reported cstate change: term changed from 0 to 1, leader changed from <none> to 476a6a56a68c4d1f82f825d8e3858063 (127.31.12.65). New cstate: current_term: 1 leader_uuid: "476a6a56a68c4d1f82f825d8e3858063" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476a6a56a68c4d1f82f825d8e3858063" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 35329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9b3cd5fcc0294fe4a8acae8034f80c56" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 46101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "190a8eb1e7244fb0899a075100cd6d10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 42231 } health_report { overall_health: UNKNOWN } } }
I20260924 15:22:25.879249 18732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.877012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40050 (local address 127.31.12.65:35329)
0924 15:22:25.877070 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:25.877072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:25.877082 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:22:25.877117 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:25.877118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:25.877137 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:25.877195 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:22:25.877315 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.877824 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.878570 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:25.878748 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:25.878797 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:25.878962 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:22:25.879044 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:22:25.879117 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:25.879165 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 15:22:25.879175 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 15:22:25.881109 18675 raft_consensus.cc:1261] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56 [term 1 FOLLOWER]: Refusing update from remote peer 476a6a56a68c4d1f82f825d8e3858063: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:22:25.881151 18544 raft_consensus.cc:1261] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10 [term 1 FOLLOWER]: Refusing update from remote peer 476a6a56a68c4d1f82f825d8e3858063: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:22:25.881539 18748 consensus_queue.cc:1035] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "190a8eb1e7244fb0899a075100cd6d10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 42231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:22:25.881685 18743 consensus_queue.cc:1035] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b3cd5fcc0294fe4a8acae8034f80c56" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 46101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:22:25.884091 18751 mvcc.cc:205] Tried to move back new op lower bound from 7332918664725983232 to 7332918664601739264. Current Snapshot: MvccSnapshot[applied={T|T < 7332918664725983232}]
I20260924 15:22:25.884617 18752 mvcc.cc:205] Tried to move back new op lower bound from 7332918664725983232 to 7332918664601739264. Current Snapshot: MvccSnapshot[applied={T|T < 7332918664725983232}]
I20260924 15:22:25.887022 18266 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:22:25.887107 18266 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:22:25.887825 18266 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bdad8dbee2245bbb85b63180fc1103f: Sending DeleteTablet for 3 replicas of tablet bc4adb480a0c47ecabd1af4b535c3326
I20260924 15:22:25.888176 18390 tablet_service.cc:1514] Processing DeleteTablet for tablet bc4adb480a0c47ecabd1af4b535c3326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:22:25 UTC) from {username='slave'} at 127.0.0.1:40038
I20260924 15:22:25.888227 18655 tablet_service.cc:1514] Processing DeleteTablet for tablet bc4adb480a0c47ecabd1af4b535c3326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:22:25 UTC) from {username='slave'} at 127.0.0.1:45178
I20260924 15:22:25.888191 18523 tablet_service.cc:1514] Processing DeleteTablet for tablet bc4adb480a0c47ecabd1af4b535c3326 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:22:25 UTC) from {username='slave'} at 127.0.0.1:53580
I20260924 15:22:25.888466 18761 tablet_replica.cc:330] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063: stopping tablet replica
I20260924 15:22:25.888567 18761 raft_consensus.cc:2229] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:22:25.888661 18763 tablet_replica.cc:330] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10: stopping tablet replica
I20260924 15:22:25.888701 18761 raft_consensus.cc:2258] T bc4adb480a0c47ecabd1af4b535c3326 P 476a6a56a68c4d1f82f825d8e3858063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:22:25.888765 18763 raft_consensus.cc:2229] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:22:25.888759 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 18328
I20260924 15:22:25.888866 18763 raft_consensus.cc:2258] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:22:25.889489 18762 tablet_replica.cc:330] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56: stopping tablet replica
I20260924 15:22:25.889591 18762 raft_consensus.cc:2229] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:22:25.889714 18762 raft_consensus.cc:2258] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:22:25.890064 18763 ts_tablet_manager.cc:1905] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:22:25.892012 18763 ts_tablet_manager.cc:1918] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:22:25.892088 18763 log.cc:1196] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/bc4adb480a0c47ecabd1af4b535c3326
I20260924 15:22:25.892326 18763 ts_tablet_manager.cc:1939] T bc4adb480a0c47ecabd1af4b535c3326 P 190a8eb1e7244fb0899a075100cd6d10: Deleting consensus metadata
I20260924 15:22:25.892726 18252 catalog_manager.cc:4841] TS 190a8eb1e7244fb0899a075100cd6d10 (127.31.12.66:42231): tablet bc4adb480a0c47ecabd1af4b535c3326 (table test-table [id=5527b9bcc07f49289097ae1a169cf4ce]) successfully deleted
I20260924 15:22:25.892973 18762 ts_tablet_manager.cc:1905] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:22:25.894847 18762 ts_tablet_manager.cc:1918] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:22:25.894920 18762 log.cc:1196] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/bc4adb480a0c47ecabd1af4b535c3326
I20260924 15:22:25.895141 18762 ts_tablet_manager.cc:1939] T bc4adb480a0c47ecabd1af4b535c3326 P 9b3cd5fcc0294fe4a8acae8034f80c56: Deleting consensus metadata
I20260924 15:22:25.895623 18253 catalog_manager.cc:4841] TS 9b3cd5fcc0294fe4a8acae8034f80c56 (127.31.12.67:46101): tablet bc4adb480a0c47ecabd1af4b535c3326 (table test-table [id=5527b9bcc07f49289097ae1a169cf4ce]) successfully deleted
W20260924 15:22:25.897583 18252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.12.65:35329 (error 108)
I20260924 15:22:25.897842 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 18461
I20260924 15:22:25.898065 18731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:25.897944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:35329 (local address 127.0.0.1:40052)
0924 15:22:25.897992 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:25.898028 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.65:35329: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 15:22:25.898168 18252 catalog_manager.cc:4584] TS 476a6a56a68c4d1f82f825d8e3858063 (127.31.12.65:35329): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc4adb480a0c47ecabd1af4b535c3326: Network error: Client connection negotiation failed: client connection to 127.31.12.65:35329: connect: Connection refused (error 111)
I20260924 15:22:25.903244 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 18593
I20260924 15:22:25.907998 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 18235
2026-09-24T15:22:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (771 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:25 dist-test-slave-zwbb krb5kdc[18767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:25 dist-test-slave-zwbb krb5kdc[18767](info): set up 2 sockets
Sep 24 15:22:25 dist-test-slave-zwbb krb5kdc[18767](info): commencing operation
krb5kdc: starting...
W20260924 15:22:26.950605 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:27 dist-test-slave-zwbb krb5kdc[18767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:22:27.065297 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:42525
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:42525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:27.145880 18780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:27.146085 18780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:27.146106 18780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:27.147269 18780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:27.147310 18780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:27.147324 18780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:27.147336 18780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:27.147348 18780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:27.148826 18780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:42525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:42525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:27.149067 18780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:27.149279 18780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:22:27.154151 18780 server_base.cc:1034] running on GCE node
W20260924 15:22:27.154138 18786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:27.154102 18787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:27.154094 18789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:27.154688 18780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:27.154949 18780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:27.156096 18780 hybrid_clock.cc:648] HybridClock initialized: now 1790263347156072 us; error 29 us; skew 500 ppm
Sep 24 15:22:27 dist-test-slave-zwbb krb5kdc[18767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263347, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:27.158582 18780 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:22:27.159000 18780 webserver.cc:460] Webserver started at http://127.31.12.126:35999/ using document root <none> and password file <none>
I20260924 15:22:27.159221 18780 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:27.159264 18780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:27.159407 18780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:27.160386 18780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "b5f3f862590f42e5896f73846d309ea7"
format_stamp: "Formatted at 2026-09-24 15:22:27 on dist-test-slave-zwbb"
server_key: "6557b97fead368e2bf5e51b2ac087c28"
server_key_iv: "77c6ac3ef7ddbdb92167494ac5b5f23b"
server_key_version: "encryptionkey@0"
I20260924 15:22:27.160732 18780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "b5f3f862590f42e5896f73846d309ea7"
format_stamp: "Formatted at 2026-09-24 15:22:27 on dist-test-slave-zwbb"
server_key: "6557b97fead368e2bf5e51b2ac087c28"
server_key_iv: "77c6ac3ef7ddbdb92167494ac5b5f23b"
server_key_version: "encryptionkey@0"
I20260924 15:22:27.162036 18780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:22:27.162703 18796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:27.162889 18780 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:27.162981 18780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "b5f3f862590f42e5896f73846d309ea7"
format_stamp: "Formatted at 2026-09-24 15:22:27 on dist-test-slave-zwbb"
server_key: "6557b97fead368e2bf5e51b2ac087c28"
server_key_iv: "77c6ac3ef7ddbdb92167494ac5b5f23b"
server_key_version: "encryptionkey@0"
I20260924 15:22:27.163030 18780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:27.175100 18780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:27.177223 18780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:27.177330 18780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:27.181005 18780 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:42525
I20260924 15:22:27.181070 18848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:42525 every 8 connection(s)
I20260924 15:22:27.181383 18780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:27.182255 18849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:27.184427 18849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7: Bootstrap starting.
I20260924 15:22:27.184948 18849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:27.185184 18849 log.cc:824] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:27.185873 18849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7: No bootstrap required, opened a new log
I20260924 15:22:27.187351 18849 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f3f862590f42e5896f73846d309ea7" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 42525 } }
I20260924 15:22:27.187494 18849 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:27.187517 18849 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5f3f862590f42e5896f73846d309ea7, State: Initialized, Role: FOLLOWER
I20260924 15:22:27.187644 18849 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f3f862590f42e5896f73846d309ea7" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 42525 } }
I20260924 15:22:27.187722 18849 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:27.187747 18849 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:27.187773 18849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:27.188220 18849 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f3f862590f42e5896f73846d309ea7" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 42525 } }
I20260924 15:22:27.188299 18849 leader_election.cc:302] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5f3f862590f42e5896f73846d309ea7; no voters:
I20260924 15:22:27.188428 18849 leader_election.cc:288] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:27.188474 18852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:27.188616 18852 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7 [term 1 LEADER]: Becoming Leader. State: Replica: b5f3f862590f42e5896f73846d309ea7, State: Running, Role: LEADER
I20260924 15:22:27.188704 18849 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:27.188750 18852 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f3f862590f42e5896f73846d309ea7" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 42525 } }
I20260924 15:22:27.189124 18854 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5f3f862590f42e5896f73846d309ea7. Latest consensus state: current_term: 1 leader_uuid: "b5f3f862590f42e5896f73846d309ea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f3f862590f42e5896f73846d309ea7" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 42525 } } }
I20260924 15:22:27.189257 18853 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5f3f862590f42e5896f73846d309ea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5f3f862590f42e5896f73846d309ea7" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 42525 } } }
I20260924 15:22:27.189292 18854 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:27.189350 18853 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:27.189734 18862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:27.190078 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 18780
I20260924 15:22:27.190187 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:27.190274 18862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:27.190426 31793 external_mini_cluster.cc:1428] Setting key 4f7d9355c0f942c895747b9886225602
I20260924 15:22:27.192214 18862 catalog_manager.cc:1347] Generated new cluster ID: 9f09b43f2e80446399349cfbe797d31d
I20260924 15:22:27.192265 18862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:22:27 dist-test-slave-zwbb krb5kdc[18767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:22:27.200551 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:27.191074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33676 (local address 127.31.12.126:42525)
0924 15:22:27.191433 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:27.191441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:27.191476 (+ 35us) server_negotiation.cc:407] Connection header received
0924 15:22:27.191583 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:27.191590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:27.191644 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:27.191783 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:27.192563 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.193542 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.194309 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.194748 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.194878 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:27.197254 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:27.197258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:27.197259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:27.197270 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:27.199011 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:27.199399 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:27.199415 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:27.199415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:27.199517 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:27.199731 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:27.199732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:27.199732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:27.199975 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:27.200147 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:27.200216 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 15:22:27.200368 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":79,"threads_started":1}
I20260924 15:22:27.216106 18862 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:27.216632 18862 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:27.221655 18862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7: Generated new TSK 0
I20260924 15:22:27.221846 18862 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.31.12.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
Entry for principal kudu/127.31.12.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab.
I20260924 15:22:27.259987 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:42525
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:27.351927 18877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:27.352100 18877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:27.352121 18877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:27.353196 18877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:27.353255 18877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:27.353281 18877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:22:27.354820 18877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.65.keytab
--principal=kudu/127.31.12.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:42525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:27.355059 18877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:27.355269 18877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:27.359934 18883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:27.359961 18882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:27.359962 18885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:27.360090 18877 server_base.cc:1034] running on GCE node
I20260924 15:22:27.360344 18877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:27.360576 18877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:27.361733 18877 hybrid_clock.cc:648] HybridClock initialized: now 1790263347361716 us; error 45 us; skew 500 ppm
Sep 24 15:22:27 dist-test-slave-zwbb krb5kdc[18767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263347, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:27.364181 18877 init.cc:377] Logged in from keytab as kudu/127.31.12.65@KRBTEST.COM (short username kudu)
I20260924 15:22:27.364629 18877 webserver.cc:460] Webserver started at http://127.31.12.65:37661/ using document root <none> and password file <none>
I20260924 15:22:27.364871 18877 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:27.364946 18877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:27.365089 18877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:27.366159 18877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "d94e1c1bfbb4445d9757f2acb37b5629"
format_stamp: "Formatted at 2026-09-24 15:22:27 on dist-test-slave-zwbb"
server_key: "e784cf080e34428cb5e6341a1094eb8e"
server_key_iv: "b8a8745c3cb40563cf27bd97af91ba24"
server_key_version: "encryptionkey@0"
I20260924 15:22:27.366545 18877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "d94e1c1bfbb4445d9757f2acb37b5629"
format_stamp: "Formatted at 2026-09-24 15:22:27 on dist-test-slave-zwbb"
server_key: "e784cf080e34428cb5e6341a1094eb8e"
server_key_iv: "b8a8745c3cb40563cf27bd97af91ba24"
server_key_version: "encryptionkey@0"
I20260924 15:22:27.367843 18877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:22:27.368520 18892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:27.368680 18877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:27.368820 18877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "d94e1c1bfbb4445d9757f2acb37b5629"
format_stamp: "Formatted at 2026-09-24 15:22:27 on dist-test-slave-zwbb"
server_key: "e784cf080e34428cb5e6341a1094eb8e"
server_key_iv: "b8a8745c3cb40563cf27bd97af91ba24"
server_key_version: "encryptionkey@0"
I20260924 15:22:27.368918 18877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:27.377090 18877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:27.379206 18877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:27.379352 18877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:27.379546 18877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:27.379817 18877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:22:27.379866 18877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:27.379910 18877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:22:27.379940 18877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:27.385484 18877 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:42107
I20260924 15:22:27.385519 19007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:42107 every 8 connection(s)
I20260924 15:22:27.385802 18877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
Sep 24 15:22:27 dist-test-slave-zwbb krb5kdc[18767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263347, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.65@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:22:27.394835 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:27.386484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:42519 (local address 127.31.12.126:42525)
0924 15:22:27.386537 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:27.386539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:27.386707 (+ 168us) server_negotiation.cc:407] Connection header received
0924 15:22:27.387153 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:27.387154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:27.387165 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:27.387215 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:27.388067 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.388565 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.389729 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.389933 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.389987 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:27.392046 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:27.392048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:27.392048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:27.392054 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:27.393424 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:27.393864 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:27.393865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:27.393865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:27.393931 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:27.394190 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:27.394191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:27.394192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:27.394310 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:27.394397 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:27.394654 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 15:22:27.394744 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 15:22:27.395015 19010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:27.386429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:42525 (local address 127.31.12.65:42519)
0924 15:22:27.386589 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:27.386629 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:27.387074 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:27.387272 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:27.387275 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:27.387807 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:27.387994 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:27.387998 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.388676 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.388677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:27.389654 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:27.389657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.390009 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.390010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:27.390117 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:27.390621 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:27.390629 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:27.391962 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:27.393550 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:27.393558 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:27.393560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:27.393784 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:27.393999 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:27.394000 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:27.394000 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:27.394115 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:27.394381 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:27.394388 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:27.394524 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 15:22:27.394630 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 15:22:27.394779 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260924 15:22:27.395263 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 18877
I20260924 15:22:27.395363 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:22:27.395610 31793 external_mini_cluster.cc:1428] Setting key cdaee522241e68a69fcc1e303abec1a4
I20260924 15:22:27.395602 19008 heartbeater.cc:347] Connected to a master server at 127.31.12.126:42525
I20260924 15:22:27.395704 19008 heartbeater.cc:464] Registering TS with master...
I20260924 15:22:27.395866 19008 heartbeater.cc:511] Master 127.31.12.126:42525 requested a full tablet report, sending...
I20260924 15:22:27.396395 18813 ts_manager.cc:194] Registered new tserver with Master: d94e1c1bfbb4445d9757f2acb37b5629 (127.31.12.65:42107)
I20260924 15:22:27.396988 18813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.65@KRBTEST.COM'} at 127.31.12.65:42519
WARNING: no policy specified for kudu/127.31.12.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
Entry for principal kudu/127.31.12.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab.
I20260924 15:22:27.418601 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:42525
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:27.496629 19013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:27.496817 19013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:27.496840 19013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:27.497953 19013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:27.497990 19013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:27.498015 19013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:22:27.499306 19013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.66.keytab
--principal=kudu/127.31.12.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:42525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:27.499459 19013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:27.499645 19013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:27.504319 19021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:27.504312 19019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:27.504340 19013 server_base.cc:1034] running on GCE node
W20260924 15:22:27.504311 19018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:27.504740 19013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:27.504989 19013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:27.506129 19013 hybrid_clock.cc:648] HybridClock initialized: now 1790263347506113 us; error 32 us; skew 500 ppm
Sep 24 15:22:27 dist-test-slave-zwbb krb5kdc[18767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263347, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:27.508574 19013 init.cc:377] Logged in from keytab as kudu/127.31.12.66@KRBTEST.COM (short username kudu)
I20260924 15:22:27.508996 19013 webserver.cc:460] Webserver started at http://127.31.12.66:44365/ using document root <none> and password file <none>
I20260924 15:22:27.509209 19013 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:27.509253 19013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:27.509392 19013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:27.510426 19013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "82c82e30b04241c3a5e32075b190ad10"
format_stamp: "Formatted at 2026-09-24 15:22:27 on dist-test-slave-zwbb"
server_key: "ea5058bce2f5d18fa91c89c317570a18"
server_key_iv: "4d194ab7a4ccb590e03edca92f15d0ad"
server_key_version: "encryptionkey@0"
I20260924 15:22:27.510782 19013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "82c82e30b04241c3a5e32075b190ad10"
format_stamp: "Formatted at 2026-09-24 15:22:27 on dist-test-slave-zwbb"
server_key: "ea5058bce2f5d18fa91c89c317570a18"
server_key_iv: "4d194ab7a4ccb590e03edca92f15d0ad"
server_key_version: "encryptionkey@0"
I20260924 15:22:27.512120 19013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:22:27.512825 19028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:27.513021 19013 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:27.513105 19013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "82c82e30b04241c3a5e32075b190ad10"
format_stamp: "Formatted at 2026-09-24 15:22:27 on dist-test-slave-zwbb"
server_key: "ea5058bce2f5d18fa91c89c317570a18"
server_key_iv: "4d194ab7a4ccb590e03edca92f15d0ad"
server_key_version: "encryptionkey@0"
I20260924 15:22:27.513178 19013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:27.526382 19013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:27.528192 19013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:27.528357 19013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:27.528584 19013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:27.528898 19013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:22:27.528950 19013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:27.528982 19013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:22:27.529012 19013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:27.534799 19013 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:34883
I20260924 15:22:27.534859 19143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:34883 every 8 connection(s)
I20260924 15:22:27.535149 19013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
Sep 24 15:22:27 dist-test-slave-zwbb krb5kdc[18767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263347, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.66@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:22:27.542817 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 19013
I20260924 15:22:27.542958 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:22:27.543196 31793 external_mini_cluster.cc:1428] Setting key c07a7296c8dffba58336a3e93d7d2032
I20260924 15:22:27.544013 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:27.535872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:60295 (local address 127.31.12.126:42525)
0924 15:22:27.535952 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:27.535955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:27.536084 (+ 129us) server_negotiation.cc:407] Connection header received
0924 15:22:27.536521 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:27.536522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:27.536531 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:27.536590 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:27.537492 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.537987 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.538941 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.539126 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.539178 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:27.541138 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:27.541139 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:27.541140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:27.541146 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:27.542547 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:27.542916 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:27.542917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:27.542918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:27.542985 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:27.543345 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:27.543347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:27.543347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:27.543501 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:27.543590 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:27.543854 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 15:22:27.543942 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 15:22:27.544245 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:27.535809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:42525 (local address 127.31.12.66:60295)
0924 15:22:27.535961 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:27.535999 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:27.536440 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:27.536643 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:27.536646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:27.537183 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:27.537372 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:27.537376 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.538069 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.538070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:27.538850 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:27.538852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.539197 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.539197 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:27.539289 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:27.539692 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:27.539700 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:27.541055 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:27.542612 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:27.542617 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:27.542619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:27.542820 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:27.543078 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:27.543079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:27.543079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:27.543223 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:27.543582 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:27.543588 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:27.543696 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 15:22:27.543821 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 15:22:27.543992 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260924 15:22:27.544696 19144 heartbeater.cc:347] Connected to a master server at 127.31.12.126:42525
I20260924 15:22:27.544793 19144 heartbeater.cc:464] Registering TS with master...
I20260924 15:22:27.544989 19144 heartbeater.cc:511] Master 127.31.12.126:42525 requested a full tablet report, sending...
I20260924 15:22:27.545341 18813 ts_manager.cc:194] Registered new tserver with Master: 82c82e30b04241c3a5e32075b190ad10 (127.31.12.66:34883)
I20260924 15:22:27.545935 18813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.66@KRBTEST.COM'} at 127.31.12.66:60295
WARNING: no policy specified for kudu/127.31.12.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
Entry for principal kudu/127.31.12.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab.
I20260924 15:22:27.568281 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:42525
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:44945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:27.651211 19149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:27.651430 19149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:27.651466 19149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:27.652575 19149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:27.652649 19149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:27.652691 19149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:22:27.654192 19149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:44945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.67.keytab
--principal=kudu/127.31.12.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:42525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:27.654445 19149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:27.654680 19149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:27.659546 19154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:27.659533 19149 server_base.cc:1034] running on GCE node
W20260924 15:22:27.659528 19155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:27.659516 19157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:27.659950 19149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:27.660183 19149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:27.661339 19149 hybrid_clock.cc:648] HybridClock initialized: now 1790263347661333 us; error 37 us; skew 500 ppm
Sep 24 15:22:27 dist-test-slave-zwbb krb5kdc[18767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263347, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:27.663858 19149 init.cc:377] Logged in from keytab as kudu/127.31.12.67@KRBTEST.COM (short username kudu)
I20260924 15:22:27.664234 19149 webserver.cc:460] Webserver started at http://127.31.12.67:33459/ using document root <none> and password file <none>
I20260924 15:22:27.664456 19149 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:27.664501 19149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:27.664640 19149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:27.665702 19149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "684e5e4a80834b0c9a89668641155e8b"
format_stamp: "Formatted at 2026-09-24 15:22:27 on dist-test-slave-zwbb"
server_key: "b84466a3e0470afcedcf2b3d12e9c37b"
server_key_iv: "04dca454fdd827f61b2761c2f5b073f3"
server_key_version: "encryptionkey@0"
I20260924 15:22:27.666059 19149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "684e5e4a80834b0c9a89668641155e8b"
format_stamp: "Formatted at 2026-09-24 15:22:27 on dist-test-slave-zwbb"
server_key: "b84466a3e0470afcedcf2b3d12e9c37b"
server_key_iv: "04dca454fdd827f61b2761c2f5b073f3"
server_key_version: "encryptionkey@0"
I20260924 15:22:27.667333 19149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:22:27.668061 19164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:27.668238 19149 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:27.668340 19149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "684e5e4a80834b0c9a89668641155e8b"
format_stamp: "Formatted at 2026-09-24 15:22:27 on dist-test-slave-zwbb"
server_key: "b84466a3e0470afcedcf2b3d12e9c37b"
server_key_iv: "04dca454fdd827f61b2761c2f5b073f3"
server_key_version: "encryptionkey@0"
I20260924 15:22:27.668414 19149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:27.700577 19149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:27.702595 19149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:27.702752 19149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:27.703013 19149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:27.703325 19149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:22:27.703380 19149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:27.703415 19149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:22:27.703460 19149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:27.709035 19149 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:41181
I20260924 15:22:27.709107 19279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:41181 every 8 connection(s)
I20260924 15:22:27.709473 19149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:22:27.712841 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 19149
I20260924 15:22:27.712914 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:22:27.713094 31793 external_mini_cluster.cc:1428] Setting key 926e4c89ca6d20d6c7e5011738c3e951
Sep 24 15:22:27 dist-test-slave-zwbb krb5kdc[18767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263347, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.67@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:22:27.718138 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:27.710194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:38439 (local address 127.31.12.126:42525)
0924 15:22:27.710237 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:27.710238 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:27.710469 (+ 231us) server_negotiation.cc:407] Connection header received
0924 15:22:27.710942 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:27.710943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:27.710952 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:27.710996 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:27.711841 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.712319 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.713394 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.713714 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.713781 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:27.715826 (+ 2045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:27.715827 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:27.715827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:27.715833 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:27.716974 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:27.717318 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:27.717319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:27.717319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:27.717368 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:27.717638 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:27.717638 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:27.717638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:27.717736 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:27.717787 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:27.718019 (+ 232us) server_negotiation.cc:299] Negotiation successful
0924 15:22:27.718078 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:22:27.718288 19282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:27.710156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:42525 (local address 127.31.12.67:38439)
0924 15:22:27.710320 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:27.710360 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:27.710847 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:27.711049 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:27.711053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:27.711586 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:27.711770 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:27.711774 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.712404 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.712405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:27.713318 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:27.713321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.713803 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.713805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:27.713950 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:27.714380 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:27.714388 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:27.715750 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:27.717039 (+ 1289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:27.717045 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:27.717046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:27.717254 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:27.717428 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:27.717428 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:27.717429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:27.717579 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:27.717793 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:27.717797 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:27.717911 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 15:22:27.717991 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 15:22:27.718103 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260924 15:22:27.718777 19280 heartbeater.cc:347] Connected to a master server at 127.31.12.126:42525
I20260924 15:22:27.718852 19280 heartbeater.cc:464] Registering TS with master...
I20260924 15:22:27.719066 19280 heartbeater.cc:511] Master 127.31.12.126:42525 requested a full tablet report, sending...
I20260924 15:22:27.719439 18813 ts_manager.cc:194] Registered new tserver with Master: 684e5e4a80834b0c9a89668641155e8b (127.31.12.67:41181)
I20260924 15:22:27.719758 18813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.31.12.67@KRBTEST.COM'} at 127.31.12.67:38439
I20260924 15:22:27.724712 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:22:27.732121 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:27.726215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.31.12.126:42525)
0924 15:22:27.726286 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:27.726288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:27.726358 (+ 70us) server_negotiation.cc:407] Connection header received
0924 15:22:27.726433 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:27.726434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:27.726443 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:27.726491 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:27.727208 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.727671 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.728483 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.728668 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.728751 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:27.729894 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:27.729895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:27.729895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:27.729901 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:27.730991 (+ 1090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:27.731346 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:27.731347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:27.731348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:27.731405 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:27.731621 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:27.731622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:27.731622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:27.731755 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:27.731873 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:27.731965 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 15:22:27.732035 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 15:22:27.733704 18813 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:22:27.735258 18813 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:22:27.741652 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:27.737602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:41181 (local address 127.0.0.1:39510)
0924 15:22:27.737723 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:27.737733 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:27.737770 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:27.738080 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:27.738082 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:27.738084 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:27.738161 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:27.738163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.739198 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.739199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:27.740410 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:27.740413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.741301 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.741302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:27.741390 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:27.741404 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 15:22:27.741515 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 15:22:27.741584 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260924 15:22:27.741658 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:27.737779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.31.12.67:41181)
0924 15:22:27.737928 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:27.737932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:27.737939 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:22:27.737966 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:27.737967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:27.737979 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:27.738064 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:27.738246 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.739099 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.740499 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.741209 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.741280 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:27.741332 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:27.741535 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 15:22:27.741591 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260924 15:22:27.742027 19292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:27.737337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:42107 (local address 127.0.0.1:33692)
0924 15:22:27.737547 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:27.737583 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:27.737646 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:27.738216 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:27.738217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:27.738219 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:27.738264 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:27.738266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.739578 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.739579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:27.740631 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:27.740633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.741771 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.741772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:27.741839 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:27.741843 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:22:27.741899 (+ 56us) client_negotiation.cc:240] Negotiation successful
0924 15:22:27.741938 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260924 15:22:27.742030 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:27.737389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:34883 (local address 127.0.0.1:50948)
0924 15:22:27.737702 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:27.737713 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:27.737763 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:27.738539 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:27.738540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:27.738542 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:27.738586 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:27.738588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.739792 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.739793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:27.740707 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:27.740711 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.741623 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.741623 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:27.741682 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:27.741686 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:22:27.741882 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 15:22:27.741923 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":1,"thread_start_us":108,"threads_started":1}
I20260924 15:22:27.742101 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:27.737525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50948 (local address 127.31.12.66:34883)
0924 15:22:27.738369 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:27.738373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:27.738382 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:22:27.738421 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:27.738423 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:27.738439 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:27.738516 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:27.738659 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.739659 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.740789 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.741532 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.741602 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:27.741649 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:27.741977 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 15:22:27.742022 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":807,"thread_start_us":68,"threads_started":1}
I20260924 15:22:27.742237 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:27.737557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33692 (local address 127.31.12.65:42107)
0924 15:22:27.737957 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:27.737961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:27.737969 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:22:27.738001 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:27.738002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:27.738013 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:27.738218 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:27.738357 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.739478 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.740726 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.741218 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.741308 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:27.741364 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:27.742124 (+ 760us) server_negotiation.cc:299] Negotiation successful
0924 15:22:27.742170 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":78,"threads_started":1}
I20260924 15:22:27.742355 19212 tablet_service.cc:1467] Processing CreateTablet for tablet 9c936f1725184525b4ddb8391d1c6eee (DEFAULT_TABLE table=test-table [id=bb6eb501db7545a8ba235d287cc7a695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:22:27.742698 19212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c936f1725184525b4ddb8391d1c6eee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:27.742749 19076 tablet_service.cc:1467] Processing CreateTablet for tablet 9c936f1725184525b4ddb8391d1c6eee (DEFAULT_TABLE table=test-table [id=bb6eb501db7545a8ba235d287cc7a695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:22:27.742990 19076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c936f1725184525b4ddb8391d1c6eee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:27.745155 19297 tablet_bootstrap.cc:492] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b: Bootstrap starting.
I20260924 15:22:27.745716 18941 tablet_service.cc:1467] Processing CreateTablet for tablet 9c936f1725184525b4ddb8391d1c6eee (DEFAULT_TABLE table=test-table [id=bb6eb501db7545a8ba235d287cc7a695]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:22:27.745940 18941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c936f1725184525b4ddb8391d1c6eee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:27.745941 19297 tablet_bootstrap.cc:654] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:27.746344 19297 log.cc:824] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:27.747078 19297 tablet_bootstrap.cc:492] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b: No bootstrap required, opened a new log
I20260924 15:22:27.747123 19297 ts_tablet_manager.cc:1397] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 15:22:27.748274 19300 tablet_bootstrap.cc:492] T 9c936f1725184525b4ddb8391d1c6eee P d94e1c1bfbb4445d9757f2acb37b5629: Bootstrap starting.
I20260924 15:22:27.748307 19297 raft_consensus.cc:348] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d94e1c1bfbb4445d9757f2acb37b5629" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42107 } } peers { permanent_uuid: "82c82e30b04241c3a5e32075b190ad10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 34883 } } peers { permanent_uuid: "684e5e4a80834b0c9a89668641155e8b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41181 } }
I20260924 15:22:27.748427 19297 raft_consensus.cc:374] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:27.748447 19297 raft_consensus.cc:729] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 684e5e4a80834b0c9a89668641155e8b, State: Initialized, Role: FOLLOWER
I20260924 15:22:27.748574 19297 consensus_queue.cc:260] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d94e1c1bfbb4445d9757f2acb37b5629" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42107 } } peers { permanent_uuid: "82c82e30b04241c3a5e32075b190ad10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 34883 } } peers { permanent_uuid: "684e5e4a80834b0c9a89668641155e8b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41181 } }
I20260924 15:22:27.748862 19298 tablet_bootstrap.cc:492] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10: Bootstrap starting.
I20260924 15:22:27.748919 19297 ts_tablet_manager.cc:1428] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:22:27.749059 19280 heartbeater.cc:503] Master 127.31.12.126:42525 was elected leader, sending a full tablet report...
I20260924 15:22:27.749164 19300 tablet_bootstrap.cc:654] T 9c936f1725184525b4ddb8391d1c6eee P d94e1c1bfbb4445d9757f2acb37b5629: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:27.749550 19298 tablet_bootstrap.cc:654] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:27.749806 19300 log.cc:824] T 9c936f1725184525b4ddb8391d1c6eee P d94e1c1bfbb4445d9757f2acb37b5629: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:27.750176 19298 log.cc:824] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:27.750648 19300 tablet_bootstrap.cc:492] T 9c936f1725184525b4ddb8391d1c6eee P d94e1c1bfbb4445d9757f2acb37b5629: No bootstrap required, opened a new log
I20260924 15:22:27.750722 19300 ts_tablet_manager.cc:1397] T 9c936f1725184525b4ddb8391d1c6eee P d94e1c1bfbb4445d9757f2acb37b5629: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260924 15:22:27.751024 19298 tablet_bootstrap.cc:492] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10: No bootstrap required, opened a new log
I20260924 15:22:27.751076 19298 ts_tablet_manager.cc:1397] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 15:22:27.752220 19300 raft_consensus.cc:348] T 9c936f1725184525b4ddb8391d1c6eee P d94e1c1bfbb4445d9757f2acb37b5629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d94e1c1bfbb4445d9757f2acb37b5629" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42107 } } peers { permanent_uuid: "82c82e30b04241c3a5e32075b190ad10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 34883 } } peers { permanent_uuid: "684e5e4a80834b0c9a89668641155e8b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41181 } }
I20260924 15:22:27.752377 19300 raft_consensus.cc:374] T 9c936f1725184525b4ddb8391d1c6eee P d94e1c1bfbb4445d9757f2acb37b5629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:27.752426 19300 raft_consensus.cc:729] T 9c936f1725184525b4ddb8391d1c6eee P d94e1c1bfbb4445d9757f2acb37b5629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d94e1c1bfbb4445d9757f2acb37b5629, State: Initialized, Role: FOLLOWER
I20260924 15:22:27.752537 19300 consensus_queue.cc:260] T 9c936f1725184525b4ddb8391d1c6eee P d94e1c1bfbb4445d9757f2acb37b5629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d94e1c1bfbb4445d9757f2acb37b5629" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42107 } } peers { permanent_uuid: "82c82e30b04241c3a5e32075b190ad10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 34883 } } peers { permanent_uuid: "684e5e4a80834b0c9a89668641155e8b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41181 } }
I20260924 15:22:27.752478 19298 raft_consensus.cc:348] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d94e1c1bfbb4445d9757f2acb37b5629" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42107 } } peers { permanent_uuid: "82c82e30b04241c3a5e32075b190ad10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 34883 } } peers { permanent_uuid: "684e5e4a80834b0c9a89668641155e8b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41181 } }
I20260924 15:22:27.752629 19298 raft_consensus.cc:374] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:27.752684 19298 raft_consensus.cc:729] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82c82e30b04241c3a5e32075b190ad10, State: Initialized, Role: FOLLOWER
I20260924 15:22:27.752833 19008 heartbeater.cc:503] Master 127.31.12.126:42525 was elected leader, sending a full tablet report...
I20260924 15:22:27.752818 19300 ts_tablet_manager.cc:1428] T 9c936f1725184525b4ddb8391d1c6eee P d94e1c1bfbb4445d9757f2acb37b5629: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 15:22:27.752825 19298 consensus_queue.cc:260] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d94e1c1bfbb4445d9757f2acb37b5629" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42107 } } peers { permanent_uuid: "82c82e30b04241c3a5e32075b190ad10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 34883 } } peers { permanent_uuid: "684e5e4a80834b0c9a89668641155e8b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41181 } }
I20260924 15:22:27.753087 19144 heartbeater.cc:503] Master 127.31.12.126:42525 was elected leader, sending a full tablet report...
I20260924 15:22:27.753345 19298 ts_tablet_manager.cc:1428] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260924 15:22:27.785660 19145 tablet.cc:2369] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:22:27.871281 19305 raft_consensus.cc:482] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:22:27.871443 19305 raft_consensus.cc:504] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d94e1c1bfbb4445d9757f2acb37b5629" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42107 } } peers { permanent_uuid: "82c82e30b04241c3a5e32075b190ad10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 34883 } } peers { permanent_uuid: "684e5e4a80834b0c9a89668641155e8b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41181 } }
I20260924 15:22:27.871918 19305 leader_election.cc:288] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d94e1c1bfbb4445d9757f2acb37b5629 (127.31.12.65:42107), 684e5e4a80834b0c9a89668641155e8b (127.31.12.67:41181)
I20260924 15:22:27.875861 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:27.872225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:48969 (local address 127.31.12.67:41181)
0924 15:22:27.872277 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:27.872279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:27.872393 (+ 114us) server_negotiation.cc:407] Connection header received
0924 15:22:27.872451 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:27.872452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:27.872460 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:27.872513 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:27.872673 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.873394 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.874927 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.875496 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.875576 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:27.875584 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:27.875759 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 15:22:27.875794 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 15:22:27.875880 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:27.872155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:52419 (local address 127.31.12.65:42107)
0924 15:22:27.872277 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:27.872279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:27.872289 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:22:27.872318 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:27.872319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:27.872329 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:27.872396 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:27.872593 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.873395 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.874923 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.875497 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.875576 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:27.875584 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:27.875756 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 15:22:27.875795 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 15:22:27.875880 19146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:27.872073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:42107 (local address 127.31.12.66:52419)
0924 15:22:27.872140 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:27.872151 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:27.872216 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:27.872416 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:27.872418 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:27.872419 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:27.872518 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:27.872524 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.873498 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.873499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:27.874821 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:27.874825 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.875583 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.875584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:27.875678 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:27.875679 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 15:22:27.875736 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:22:27.875782 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":61}
I20260924 15:22:27.875912 19306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:27.872114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:41181 (local address 127.31.12.66:48969)
0924 15:22:27.872312 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:27.872320 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:27.872396 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:27.872531 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:27.872532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:27.872533 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:27.872596 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:27.872597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.873498 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.873499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:27.874821 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:27.874825 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.875583 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.875584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:27.875673 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:27.875678 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 15:22:27.875736 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 15:22:27.875784 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":91,"thread_start_us":98,"threads_started":1}
I20260924 15:22:27.876223 18961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c936f1725184525b4ddb8391d1c6eee" candidate_uuid: "82c82e30b04241c3a5e32075b190ad10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d94e1c1bfbb4445d9757f2acb37b5629" is_pre_election: true
I20260924 15:22:27.876408 18961 raft_consensus.cc:2454] T 9c936f1725184525b4ddb8391d1c6eee P d94e1c1bfbb4445d9757f2acb37b5629 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c82e30b04241c3a5e32075b190ad10 in term 0.
I20260924 15:22:27.876358 19233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c936f1725184525b4ddb8391d1c6eee" candidate_uuid: "82c82e30b04241c3a5e32075b190ad10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "684e5e4a80834b0c9a89668641155e8b" is_pre_election: true
I20260924 15:22:27.876497 19233 raft_consensus.cc:2454] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82c82e30b04241c3a5e32075b190ad10 in term 0.
I20260924 15:22:27.876644 19032 leader_election.cc:302] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82c82e30b04241c3a5e32075b190ad10, d94e1c1bfbb4445d9757f2acb37b5629; no voters:
I20260924 15:22:27.876806 19305 raft_consensus.cc:2781] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:22:27.876890 19305 raft_consensus.cc:482] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:22:27.876924 19305 raft_consensus.cc:3037] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:27.877719 19305 raft_consensus.cc:504] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d94e1c1bfbb4445d9757f2acb37b5629" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42107 } } peers { permanent_uuid: "82c82e30b04241c3a5e32075b190ad10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 34883 } } peers { permanent_uuid: "684e5e4a80834b0c9a89668641155e8b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41181 } }
I20260924 15:22:27.877859 19305 leader_election.cc:288] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [CANDIDATE]: Term 1 election: Requested vote from peers d94e1c1bfbb4445d9757f2acb37b5629 (127.31.12.65:42107), 684e5e4a80834b0c9a89668641155e8b (127.31.12.67:41181)
I20260924 15:22:27.878051 18961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c936f1725184525b4ddb8391d1c6eee" candidate_uuid: "82c82e30b04241c3a5e32075b190ad10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d94e1c1bfbb4445d9757f2acb37b5629"
I20260924 15:22:27.878039 19233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c936f1725184525b4ddb8391d1c6eee" candidate_uuid: "82c82e30b04241c3a5e32075b190ad10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "684e5e4a80834b0c9a89668641155e8b"
I20260924 15:22:27.878122 19233 raft_consensus.cc:3037] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:27.878122 18961 raft_consensus.cc:3037] T 9c936f1725184525b4ddb8391d1c6eee P d94e1c1bfbb4445d9757f2acb37b5629 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:27.878973 19233 raft_consensus.cc:2454] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c82e30b04241c3a5e32075b190ad10 in term 1.
I20260924 15:22:27.878973 18961 raft_consensus.cc:2454] T 9c936f1725184525b4ddb8391d1c6eee P d94e1c1bfbb4445d9757f2acb37b5629 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82c82e30b04241c3a5e32075b190ad10 in term 1.
I20260924 15:22:27.879186 19030 leader_election.cc:302] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 684e5e4a80834b0c9a89668641155e8b, 82c82e30b04241c3a5e32075b190ad10; no voters:
I20260924 15:22:27.879285 19305 raft_consensus.cc:2781] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:27.879465 19305 raft_consensus.cc:686] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [term 1 LEADER]: Becoming Leader. State: Replica: 82c82e30b04241c3a5e32075b190ad10, State: Running, Role: LEADER
I20260924 15:22:27.879592 19305 consensus_queue.cc:237] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d94e1c1bfbb4445d9757f2acb37b5629" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42107 } } peers { permanent_uuid: "82c82e30b04241c3a5e32075b190ad10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 34883 } } peers { permanent_uuid: "684e5e4a80834b0c9a89668641155e8b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41181 } }
I20260924 15:22:27.880436 18813 catalog_manager.cc:5495] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82c82e30b04241c3a5e32075b190ad10 (127.31.12.66). New cstate: current_term: 1 leader_uuid: "82c82e30b04241c3a5e32075b190ad10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d94e1c1bfbb4445d9757f2acb37b5629" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82c82e30b04241c3a5e32075b190ad10" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 34883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "684e5e4a80834b0c9a89668641155e8b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41181 } health_report { overall_health: UNKNOWN } } }
W20260924 15:22:27.886500 19009 tablet.cc:2369] T 9c936f1725184525b4ddb8391d1c6eee P d94e1c1bfbb4445d9757f2acb37b5629: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:22:27.920871 19293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:27.918337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50964 (local address 127.31.12.66:34883)
0924 15:22:27.918407 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:27.918409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:27.918419 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:22:27.918509 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:27.918509 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:27.918528 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:27.918600 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:22:27.918747 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.919220 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.920051 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:27.920249 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:27.920339 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:27.920527 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:22:27.920632 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:22:27.920731 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:27.920749 (+ 18us) server_negotiation.cc:299] Negotiation successful
0924 15:22:27.920769 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 15:22:27.922811 19233 raft_consensus.cc:1261] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b [term 1 FOLLOWER]: Refusing update from remote peer 82c82e30b04241c3a5e32075b190ad10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:22:27.922807 18961 raft_consensus.cc:1261] T 9c936f1725184525b4ddb8391d1c6eee P d94e1c1bfbb4445d9757f2acb37b5629 [term 1 FOLLOWER]: Refusing update from remote peer 82c82e30b04241c3a5e32075b190ad10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:22:27.923218 19305 consensus_queue.cc:1035] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "684e5e4a80834b0c9a89668641155e8b" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 41181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:22:27.923372 19307 consensus_queue.cc:1035] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d94e1c1bfbb4445d9757f2acb37b5629" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 42107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:22:27.926015 19311 mvcc.cc:205] Tried to move back new op lower bound from 7332918673088864256 to 7332918672915410944. Current Snapshot: MvccSnapshot[applied={T|T < 7332918673088864256 or (T in {7332918673088864256})}]
I20260924 15:22:27.927986 18812 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:22:27.928102 18812 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:22:27.928865 18812 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5f3f862590f42e5896f73846d309ea7: Sending DeleteTablet for 3 replicas of tablet 9c936f1725184525b4ddb8391d1c6eee
I20260924 15:22:27.929200 18941 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c936f1725184525b4ddb8391d1c6eee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:22:27 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:33692
I20260924 15:22:27.929203 19076 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c936f1725184525b4ddb8391d1c6eee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:22:27 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:50948
I20260924 15:22:27.929435 19320 tablet_replica.cc:330] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10: stopping tablet replica
I20260924 15:22:27.929486 19212 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c936f1725184525b4ddb8391d1c6eee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:22:27 UTC) from {username='kudu', principal='kudu/127.31.12.126@KRBTEST.COM'} at 127.0.0.1:39510
I20260924 15:22:27.929560 19320 raft_consensus.cc:2229] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:22:27.929663 19320 raft_consensus.cc:2258] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:22:27.929750 19322 tablet_replica.cc:330] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b: stopping tablet replica
I20260924 15:22:27.929857 19322 raft_consensus.cc:2229] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:22:27.929975 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 18877
I20260924 15:22:27.929994 19322 raft_consensus.cc:2258] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:22:27.930979 19322 ts_tablet_manager.cc:1905] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:22:27.932614 19320 ts_tablet_manager.cc:1905] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:22:27.933082 19322 ts_tablet_manager.cc:1918] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:22:27.933225 19322 log.cc:1196] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/9c936f1725184525b4ddb8391d1c6eee
I20260924 15:22:27.933593 19322 ts_tablet_manager.cc:1939] T 9c936f1725184525b4ddb8391d1c6eee P 684e5e4a80834b0c9a89668641155e8b: Deleting consensus metadata
I20260924 15:22:27.934029 18798 catalog_manager.cc:4841] TS 684e5e4a80834b0c9a89668641155e8b (127.31.12.67:41181): tablet 9c936f1725184525b4ddb8391d1c6eee (table test-table [id=bb6eb501db7545a8ba235d287cc7a695]) successfully deleted
I20260924 15:22:27.934343 19320 ts_tablet_manager.cc:1918] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:22:27.934425 19320 log.cc:1196] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/9c936f1725184525b4ddb8391d1c6eee
I20260924 15:22:27.934755 19320 ts_tablet_manager.cc:1939] T 9c936f1725184525b4ddb8391d1c6eee P 82c82e30b04241c3a5e32075b190ad10: Deleting consensus metadata
I20260924 15:22:27.935153 18798 catalog_manager.cc:4841] TS 82c82e30b04241c3a5e32075b190ad10 (127.31.12.66:34883): tablet 9c936f1725184525b4ddb8391d1c6eee (table test-table [id=bb6eb501db7545a8ba235d287cc7a695]) successfully deleted
W20260924 15:22:27.936163 18800 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.12.65:42107 (error 108)
I20260924 15:22:27.936683 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:27.936568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:42107 (local address 127.0.0.1:33708)
0924 15:22:27.936615 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:27.936651 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.65:42107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 15:22:27.936753 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 19013
W20260924 15:22:27.936792 18800 catalog_manager.cc:4584] TS d94e1c1bfbb4445d9757f2acb37b5629 (127.31.12.65:42107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9c936f1725184525b4ddb8391d1c6eee: Network error: Client connection negotiation failed: client connection to 127.31.12.65:42107: connect: Connection refused (error 111)
I20260924 15:22:27.942994 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 19149
I20260924 15:22:27.947863 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 18780
2026-09-24T15:22:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2040 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T15:22:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:27Z Disabled control of system clock
I20260924 15:22:27.975136 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:36343
--webserver_interface=127.31.12.126
--webserver_port=0
--builtin_ntp_servers=127.31.12.84:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:36343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:22:28.053414 19327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:28.053689 19327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:28.053709 19327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:28.054804 19327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:28.054838 19327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:28.054852 19327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:28.054863 19327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:28.054875 19327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:28.056068 19327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:36343
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:36343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:28.056267 19327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:28.056485 19327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:28.061005 19335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:28.061133 19327 server_base.cc:1034] running on GCE node
W20260924 15:22:28.060997 19332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:28.061260 19333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:28.061578 19327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:28.061830 19327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:28.063004 19327 hybrid_clock.cc:648] HybridClock initialized: now 1790263348062994 us; error 38 us; skew 500 ppm
I20260924 15:22:28.064200 19327 webserver.cc:460] Webserver started at http://127.31.12.126:46777/ using document root <none> and password file <none>
I20260924 15:22:28.064455 19327 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:28.064524 19327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:28.064671 19327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:28.065788 19327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "44284241f96a4fb0a0ac43891f649cae"
format_stamp: "Formatted at 2026-09-24 15:22:28 on dist-test-slave-zwbb"
server_key: "cd8b321e79f87ca0ea2ba51edde620a0"
server_key_iv: "0ea5291d88bfe68cb8dc8ed54c707965"
server_key_version: "encryptionkey@0"
I20260924 15:22:28.066159 19327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "44284241f96a4fb0a0ac43891f649cae"
format_stamp: "Formatted at 2026-09-24 15:22:28 on dist-test-slave-zwbb"
server_key: "cd8b321e79f87ca0ea2ba51edde620a0"
server_key_iv: "0ea5291d88bfe68cb8dc8ed54c707965"
server_key_version: "encryptionkey@0"
I20260924 15:22:28.067489 19327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:22:28.068220 19341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:28.068405 19327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:28.068490 19327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "44284241f96a4fb0a0ac43891f649cae"
format_stamp: "Formatted at 2026-09-24 15:22:28 on dist-test-slave-zwbb"
server_key: "cd8b321e79f87ca0ea2ba51edde620a0"
server_key_iv: "0ea5291d88bfe68cb8dc8ed54c707965"
server_key_version: "encryptionkey@0"
I20260924 15:22:28.068562 19327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:28.092666 19327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:28.092978 19327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:28.093135 19327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:28.097385 19327 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:36343
I20260924 15:22:28.097424 19394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:36343 every 8 connection(s)
I20260924 15:22:28.097811 19327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:28.098757 19395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:28.099261 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 19327
I20260924 15:22:28.099370 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:28.099598 31793 external_mini_cluster.cc:1428] Setting key e7a1183453d2568ac0018f34f7cc0a8a
I20260924 15:22:28.101665 19395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae: Bootstrap starting.
I20260924 15:22:28.102284 19395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:28.102607 19395 log.cc:824] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:28.103322 19395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae: No bootstrap required, opened a new log
I20260924 15:22:28.104991 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:28.100113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.31.12.126:36343)
0924 15:22:28.100291 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:28.100299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:28.100324 (+ 25us) server_negotiation.cc:407] Connection header received
0924 15:22:28.100726 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:28.100732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:28.100787 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:28.100910 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:28.101136 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.102076 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.103140 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.103574 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.103663 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:28.104309 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:28.104312 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:28.104312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:28.104480 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:28.104510 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:28.104520 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:28.104570 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:28.104684 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 15:22:28.104776 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
I20260924 15:22:28.105701 19395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44284241f96a4fb0a0ac43891f649cae" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 36343 } }
I20260924 15:22:28.105868 19395 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:28.105918 19395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44284241f96a4fb0a0ac43891f649cae, State: Initialized, Role: FOLLOWER
I20260924 15:22:28.106055 19395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44284241f96a4fb0a0ac43891f649cae" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 36343 } }
I20260924 15:22:28.106148 19395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:28.106187 19395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:28.106240 19395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:28.106745 19395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44284241f96a4fb0a0ac43891f649cae" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 36343 } }
I20260924 15:22:28.106827 19395 leader_election.cc:302] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44284241f96a4fb0a0ac43891f649cae; no voters:
I20260924 15:22:28.106956 19395 leader_election.cc:288] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:28.106997 19400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:28.107149 19400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae [term 1 LEADER]: Becoming Leader. State: Replica: 44284241f96a4fb0a0ac43891f649cae, State: Running, Role: LEADER
I20260924 15:22:28.107239 19395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:28.107266 19400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44284241f96a4fb0a0ac43891f649cae" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 36343 } }
I20260924 15:22:28.107690 19401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44284241f96a4fb0a0ac43891f649cae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44284241f96a4fb0a0ac43891f649cae" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 36343 } } }
I20260924 15:22:28.107718 19402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44284241f96a4fb0a0ac43891f649cae. Latest consensus state: current_term: 1 leader_uuid: "44284241f96a4fb0a0ac43891f649cae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44284241f96a4fb0a0ac43891f649cae" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 36343 } } }
I20260924 15:22:28.107873 19401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:28.107888 19402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:28.108227 19409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:28.108783 19409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:28.110299 19409 catalog_manager.cc:1347] Generated new cluster ID: ac80ea5284924e2ab7e8cbafb4b2dab7
I20260924 15:22:28.110358 19409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:28.121249 19409 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:28.121949 19409 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:28.125783 19409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae: Generated new TSK 0
I20260924 15:22:28.125972 19409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:28.131105 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.65:0
--local_ip_for_outbound_sockets=127.31.12.65
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:36343
--builtin_ntp_servers=127.31.12.84:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 15:22:28.221985 19419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:28.222193 19419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:28.222213 19419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:28.223354 19419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:28.223394 19419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:28.223420 19419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.65
I20260924 15:22:28.224751 19419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.31.12.65
--webserver_port=0
--tserver_master_addrs=127.31.12.126:36343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.65
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:28.225014 19419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:28.225318 19419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:28.229931 19427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:28.230082 19424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:28.229996 19419 server_base.cc:1034] running on GCE node
W20260924 15:22:28.229964 19425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:28.230393 19419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:28.230654 19419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:28.231810 19419 hybrid_clock.cc:648] HybridClock initialized: now 1790263348231792 us; error 36 us; skew 500 ppm
I20260924 15:22:28.233038 19419 webserver.cc:460] Webserver started at http://127.31.12.65:34317/ using document root <none> and password file <none>
I20260924 15:22:28.233320 19419 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:28.233367 19419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:28.233515 19419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:28.234540 19419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-0/data/instance:
uuid: "962f127e5e914b2da82dac5549e512bc"
format_stamp: "Formatted at 2026-09-24 15:22:28 on dist-test-slave-zwbb"
server_key: "cd4b34852524310fce9f57bd464a4697"
server_key_iv: "4470265542928fdff57740a3b048c7a8"
server_key_version: "encryptionkey@0"
I20260924 15:22:28.234879 19419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance:
uuid: "962f127e5e914b2da82dac5549e512bc"
format_stamp: "Formatted at 2026-09-24 15:22:28 on dist-test-slave-zwbb"
server_key: "cd4b34852524310fce9f57bd464a4697"
server_key_iv: "4470265542928fdff57740a3b048c7a8"
server_key_version: "encryptionkey@0"
I20260924 15:22:28.236162 19419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:22:28.236930 19433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:28.237106 19419 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:22:28.237195 19419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-0/wal
uuid: "962f127e5e914b2da82dac5549e512bc"
format_stamp: "Formatted at 2026-09-24 15:22:28 on dist-test-slave-zwbb"
server_key: "cd4b34852524310fce9f57bd464a4697"
server_key_iv: "4470265542928fdff57740a3b048c7a8"
server_key_version: "encryptionkey@0"
I20260924 15:22:28.237282 19419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:28.243804 19419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:28.244083 19419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:28.244230 19419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:28.244504 19419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:28.244813 19419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:22:28.244872 19419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:28.244906 19419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:22:28.244935 19419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:28.250540 19419 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.65:41025
I20260924 15:22:28.250586 19548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.65:41025 every 8 connection(s)
I20260924 15:22:28.250927 19419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-0/data/info.pb
I20260924 15:22:28.255874 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:28.251653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.65:56499 (local address 127.31.12.126:36343)
0924 15:22:28.251712 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:28.251719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:28.251964 (+ 245us) server_negotiation.cc:407] Connection header received
0924 15:22:28.252424 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:28.252425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:28.252437 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:28.252490 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:28.253100 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.253644 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.254615 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.254802 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.254892 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:28.255628 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:28.255630 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:28.255630 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:28.255634 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:28.255650 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:28.255653 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:28.255706 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:28.255792 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 15:22:28.255817 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 15:22:28.256109 19551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:28.251648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:36343 (local address 127.31.12.65:56499)
0924 15:22:28.251819 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:28.251850 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:28.252329 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:28.252538 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:28.252541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:28.252549 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:28.253026 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:28.253030 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.253721 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.253722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:28.254516 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:28.254518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.254915 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.254916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:28.255017 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:28.255482 (+ 465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:22:28.255483 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:28.255509 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:22:28.255529 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:28.255716 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:28.255720 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:28.255722 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:22:28.255771 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 15:22:28.255843 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260924 15:22:28.256680 19549 heartbeater.cc:347] Connected to a master server at 127.31.12.126:36343
I20260924 15:22:28.256696 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 19419
I20260924 15:22:28.256788 19549 heartbeater.cc:464] Registering TS with master...
I20260924 15:22:28.256806 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-0/wal/instance
I20260924 15:22:28.256989 19549 heartbeater.cc:511] Master 127.31.12.126:36343 requested a full tablet report, sending...
I20260924 15:22:28.257030 31793 external_mini_cluster.cc:1428] Setting key e7611eaf0f0e1b25e4b57d976c606cbd
I20260924 15:22:28.257630 19357 ts_manager.cc:194] Registered new tserver with Master: 962f127e5e914b2da82dac5549e512bc (127.31.12.65:41025)
I20260924 15:22:28.258035 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.66:0
--local_ip_for_outbound_sockets=127.31.12.66
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:36343
--builtin_ntp_servers=127.31.12.84:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 15:22:28.258543 19357 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.12.65:56499
W20260924 15:22:28.338625 19552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:28.338812 19552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:28.338833 19552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:28.339960 19552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:28.339998 19552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:28.340023 19552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.66
I20260924 15:22:28.341392 19552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.31.12.66
--webserver_port=0
--tserver_master_addrs=127.31.12.126:36343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.66
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:28.341625 19552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:28.341908 19552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:28.346836 19558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:28.346824 19560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:28.346869 19557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:28.347142 19552 server_base.cc:1034] running on GCE node
I20260924 15:22:28.347374 19552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:28.347625 19552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:28.348784 19552 hybrid_clock.cc:648] HybridClock initialized: now 1790263348348772 us; error 25 us; skew 500 ppm
I20260924 15:22:28.349996 19552 webserver.cc:460] Webserver started at http://127.31.12.66:35441/ using document root <none> and password file <none>
I20260924 15:22:28.350229 19552 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:28.350296 19552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:28.350469 19552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:28.351466 19552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-1/data/instance:
uuid: "f76337a949ae4cbda4bb81d6c95c8f2e"
format_stamp: "Formatted at 2026-09-24 15:22:28 on dist-test-slave-zwbb"
server_key: "d40b01721de2c0cbfddc54de8497d963"
server_key_iv: "0d77dfe032c8154fce593ab87660a85d"
server_key_version: "encryptionkey@0"
I20260924 15:22:28.351819 19552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance:
uuid: "f76337a949ae4cbda4bb81d6c95c8f2e"
format_stamp: "Formatted at 2026-09-24 15:22:28 on dist-test-slave-zwbb"
server_key: "d40b01721de2c0cbfddc54de8497d963"
server_key_iv: "0d77dfe032c8154fce593ab87660a85d"
server_key_version: "encryptionkey@0"
I20260924 15:22:28.353292 19552 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:22:28.354022 19566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:28.354197 19552 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:28.354291 19552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-1/wal
uuid: "f76337a949ae4cbda4bb81d6c95c8f2e"
format_stamp: "Formatted at 2026-09-24 15:22:28 on dist-test-slave-zwbb"
server_key: "d40b01721de2c0cbfddc54de8497d963"
server_key_iv: "0d77dfe032c8154fce593ab87660a85d"
server_key_version: "encryptionkey@0"
I20260924 15:22:28.354378 19552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:28.361570 19552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:28.361822 19552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:28.361967 19552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:28.362206 19552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:28.362535 19552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:22:28.362591 19552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:28.362628 19552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:22:28.362654 19552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:28.368430 19552 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.66:44059
I20260924 15:22:28.368468 19682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.66:44059 every 8 connection(s)
I20260924 15:22:28.368795 19552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-1/data/info.pb
I20260924 15:22:28.373034 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 19552
I20260924 15:22:28.373159 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-1/wal/instance
I20260924 15:22:28.373371 31793 external_mini_cluster.cc:1428] Setting key fe212b5837c8eae1d7f67ef4aebdf349
I20260924 15:22:28.374004 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:28.369530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:51017 (local address 127.31.12.126:36343)
0924 15:22:28.369587 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:28.369590 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:28.369853 (+ 263us) server_negotiation.cc:407] Connection header received
0924 15:22:28.370313 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:28.370314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:28.370325 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:28.370390 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:28.371078 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.371630 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.372586 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.372766 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.372819 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:28.373739 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:28.373740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:28.373740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:28.373745 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:28.373763 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:28.373767 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:28.373829 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:28.373921 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 15:22:28.373947 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:22:28.374214 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:28.369552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:36343 (local address 127.31.12.66:51017)
0924 15:22:28.369742 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:28.369772 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:28.370194 (+ 422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:28.370446 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:28.370449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:28.370458 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:28.371006 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:28.371011 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.371707 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.371720 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:28.372494 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:28.372496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.372840 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.372840 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:28.372960 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:28.373591 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:22:28.373591 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:28.373630 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:22:28.373658 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:28.373837 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:28.373843 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:28.373846 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:22:28.373918 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 15:22:28.374016 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260924 15:22:28.374663 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.31.12.67:0
--local_ip_for_outbound_sockets=127.31.12.67
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:36343
--builtin_ntp_servers=127.31.12.84:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 15:22:28.374913 19683 heartbeater.cc:347] Connected to a master server at 127.31.12.126:36343
I20260924 15:22:28.375000 19683 heartbeater.cc:464] Registering TS with master...
I20260924 15:22:28.375203 19683 heartbeater.cc:511] Master 127.31.12.126:36343 requested a full tablet report, sending...
I20260924 15:22:28.375571 19357 ts_manager.cc:194] Registered new tserver with Master: f76337a949ae4cbda4bb81d6c95c8f2e (127.31.12.66:44059)
I20260924 15:22:28.375945 19357 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.12.66:51017
W20260924 15:22:28.454819 19686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:28.455034 19686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:28.455070 19686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:28.456159 19686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:28.456218 19686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:28.456256 19686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.31.12.67
I20260924 15:22:28.457716 19686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.31.12.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.31.12.67
--webserver_port=0
--tserver_master_addrs=127.31.12.126:36343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.31.12.67
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:28.457950 19686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:28.458245 19686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:28.463008 19692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:28.462992 19691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:28.463128 19686 server_base.cc:1034] running on GCE node
W20260924 15:22:28.462994 19694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:28.463469 19686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:28.463694 19686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:28.464828 19686 hybrid_clock.cc:648] HybridClock initialized: now 1790263348464818 us; error 27 us; skew 500 ppm
I20260924 15:22:28.466001 19686 webserver.cc:460] Webserver started at http://127.31.12.67:36725/ using document root <none> and password file <none>
I20260924 15:22:28.466199 19686 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:28.466238 19686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:28.466324 19686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:28.467232 19686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-2/data/instance:
uuid: "426e2483c5a84543ab7c4feb99460578"
format_stamp: "Formatted at 2026-09-24 15:22:28 on dist-test-slave-zwbb"
server_key: "67527c6ccf1904a8ae28dc7c9b50ed4f"
server_key_iv: "175671fe6630eb285a28945a140ab0c2"
server_key_version: "encryptionkey@0"
I20260924 15:22:28.467532 19686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance:
uuid: "426e2483c5a84543ab7c4feb99460578"
format_stamp: "Formatted at 2026-09-24 15:22:28 on dist-test-slave-zwbb"
server_key: "67527c6ccf1904a8ae28dc7c9b50ed4f"
server_key_iv: "175671fe6630eb285a28945a140ab0c2"
server_key_version: "encryptionkey@0"
I20260924 15:22:28.468679 19686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:22:28.469302 19700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:28.469523 19686 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:28.469594 19686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-2/wal
uuid: "426e2483c5a84543ab7c4feb99460578"
format_stamp: "Formatted at 2026-09-24 15:22:28 on dist-test-slave-zwbb"
server_key: "67527c6ccf1904a8ae28dc7c9b50ed4f"
server_key_iv: "175671fe6630eb285a28945a140ab0c2"
server_key_version: "encryptionkey@0"
I20260924 15:22:28.469645 19686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:28.476927 19686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:28.477149 19686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:28.477238 19686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:28.477375 19686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 15:22:28.477718 19686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 15:22:28.477769 19686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:28.477805 19686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 15:22:28.477855 19686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:28.483489 19686 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.67:38927
I20260924 15:22:28.483546 19815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.67:38927 every 8 connection(s)
I20260924 15:22:28.483821 19686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-2/data/info.pb
I20260924 15:22:28.488830 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:28.484451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.67:36707 (local address 127.31.12.126:36343)
0924 15:22:28.484545 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:28.484547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:28.484764 (+ 217us) server_negotiation.cc:407] Connection header received
0924 15:22:28.485280 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:28.485281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:28.485294 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:28.485357 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:28.486056 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.486563 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.487532 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.487763 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.487854 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:28.488564 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:28.488565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:28.488565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:28.488570 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:28.488584 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:28.488588 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:28.488636 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:28.488726 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 15:22:28.488754 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 15:22:28.489091 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:28.484458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:36343 (local address 127.31.12.67:36707)
0924 15:22:28.484653 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:28.484683 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:28.485171 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:28.485428 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:28.485431 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:28.485475 (+ 44us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:28.485961 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:28.485966 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.486660 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.486661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:28.487419 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:28.487421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.487866 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.487867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:28.487997 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:28.488429 (+ 432us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 15:22:28.488429 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:28.488455 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 15:22:28.488475 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:28.488636 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:28.488641 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:28.488643 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 15:22:28.488730 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 15:22:28.488823 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
I20260924 15:22:28.489600 19816 heartbeater.cc:347] Connected to a master server at 127.31.12.126:36343
I20260924 15:22:28.489710 19816 heartbeater.cc:464] Registering TS with master...
I20260924 15:22:28.489840 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 19686
I20260924 15:22:28.489905 19816 heartbeater.cc:511] Master 127.31.12.126:36343 requested a full tablet report, sending...
I20260924 15:22:28.489965 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-2/wal/instance
I20260924 15:22:28.490178 31793 external_mini_cluster.cc:1428] Setting key 4d785646e5332e828402f656b17ac765
I20260924 15:22:28.490296 19357 ts_manager.cc:194] Registered new tserver with Master: 426e2483c5a84543ab7c4feb99460578 (127.31.12.67:38927)
I20260924 15:22:28.490723 19357 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.31.12.67:36707
I20260924 15:22:28.490964 31793 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 15:22:28.495214 19398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:28.492253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46098 (local address 127.31.12.126:36343)
0924 15:22:28.492284 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:28.492286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:28.492434 (+ 148us) server_negotiation.cc:407] Connection header received
0924 15:22:28.492504 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:28.492504 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:28.492513 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:28.492552 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:28.492745 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.493177 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.493986 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.494340 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.494429 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:28.494989 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:28.494990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:28.494991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:28.494995 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:28.495007 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:28.495010 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:28.495064 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:28.495120 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 15:22:28.495138 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:22:28.496567 19358 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 15:22:28.498135 19358 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 15:22:28.504236 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:28.500335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:41025 (local address 127.0.0.1:57892)
0924 15:22:28.500481 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:28.500490 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:28.500526 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:28.501131 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:28.501132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:28.501134 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:28.501182 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:28.501184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.502392 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.502393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:28.503140 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:28.503142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.503952 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.503953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:28.504030 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:28.504044 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 15:22:28.504116 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 15:22:28.504166 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":2,"thread_start_us":109,"threads_started":1}
I20260924 15:22:28.504243 19829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:28.500741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57892 (local address 127.31.12.65:41025)
0924 15:22:28.500966 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:28.500970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:28.500979 (+ 9us) server_negotiation.cc:407] Connection header received
0924 15:22:28.501015 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:28.501018 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:28.501031 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:28.501130 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:28.501270 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.502309 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.503235 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.503857 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.503940 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:28.504020 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:28.504130 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 15:22:28.504164 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260924 15:22:28.504603 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:28.500140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.66:44059 (local address 127.0.0.1:59940)
0924 15:22:28.500240 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:28.500273 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:28.500324 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:28.500808 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:28.500815 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:28.500817 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:28.500876 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:28.500878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.502238 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.502239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:28.503465 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:28.503468 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.504360 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.504361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:28.504436 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:28.504439 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 15:22:28.504485 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 15:22:28.504533 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":4,"thread_start_us":61,"threads_started":1}
I20260924 15:22:28.504603 19825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:28.500163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:38927 (local address 127.0.0.1:35264)
0924 15:22:28.500291 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:28.500302 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:28.500349 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:28.500821 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:28.500822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:28.500825 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:28.500881 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:28.500883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.502105 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.502106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:28.503464 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:28.503468 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.504403 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.504403 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:28.504452 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:28.504456 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 15:22:28.504510 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 15:22:28.504544 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":61,"thread_start_us":84,"threads_started":1}
I20260924 15:22:28.504639 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:28.500494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35264 (local address 127.31.12.67:38927)
0924 15:22:28.500627 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:28.500631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:28.500641 (+ 10us) server_negotiation.cc:407] Connection header received
0924 15:22:28.500676 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:28.500678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:28.500692 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:28.500797 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:28.500968 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.502019 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.503569 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.504291 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.504410 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:28.504500 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:28.504524 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 15:22:28.504553 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260924 15:22:28.505328 19749 tablet_service.cc:1467] Processing CreateTablet for tablet 7e195afc8ab143228e0d3f4f0a92965f (DEFAULT_TABLE table=test-table [id=ac0d6587cd0e4eb1b895dd7073acb1bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:22:28.505333 19481 tablet_service.cc:1467] Processing CreateTablet for tablet 7e195afc8ab143228e0d3f4f0a92965f (DEFAULT_TABLE table=test-table [id=ac0d6587cd0e4eb1b895dd7073acb1bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:22:28.505580 19827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:28.500324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59940 (local address 127.31.12.66:44059)
0924 15:22:28.500626 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:28.500629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:28.500637 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:22:28.500687 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:28.500689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:28.500705 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:28.500960 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:28.501027 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.502153 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.503561 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.504255 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.504712 (+ 457us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:28.504804 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:28.504827 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 15:22:28.504873 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":120,"threads_started":1}
I20260924 15:22:28.505683 19749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e195afc8ab143228e0d3f4f0a92965f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:28.505683 19481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e195afc8ab143228e0d3f4f0a92965f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:28.508970 19831 tablet_bootstrap.cc:492] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578: Bootstrap starting.
I20260924 15:22:28.509573 19831 tablet_bootstrap.cc:654] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:28.509509 19615 tablet_service.cc:1467] Processing CreateTablet for tablet 7e195afc8ab143228e0d3f4f0a92965f (DEFAULT_TABLE table=test-table [id=ac0d6587cd0e4eb1b895dd7073acb1bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 15:22:28.509816 19615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e195afc8ab143228e0d3f4f0a92965f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:28.509876 19831 log.cc:824] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:28.510212 19830 tablet_bootstrap.cc:492] T 7e195afc8ab143228e0d3f4f0a92965f P 962f127e5e914b2da82dac5549e512bc: Bootstrap starting.
I20260924 15:22:28.510778 19830 tablet_bootstrap.cc:654] T 7e195afc8ab143228e0d3f4f0a92965f P 962f127e5e914b2da82dac5549e512bc: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:28.511058 19830 log.cc:824] T 7e195afc8ab143228e0d3f4f0a92965f P 962f127e5e914b2da82dac5549e512bc: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:28.511628 19831 tablet_bootstrap.cc:492] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578: No bootstrap required, opened a new log
I20260924 15:22:28.511720 19831 ts_tablet_manager.cc:1397] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 15:22:28.511825 19830 tablet_bootstrap.cc:492] T 7e195afc8ab143228e0d3f4f0a92965f P 962f127e5e914b2da82dac5549e512bc: No bootstrap required, opened a new log
I20260924 15:22:28.511905 19830 ts_tablet_manager.cc:1397] T 7e195afc8ab143228e0d3f4f0a92965f P 962f127e5e914b2da82dac5549e512bc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 15:22:28.512758 19834 tablet_bootstrap.cc:492] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e: Bootstrap starting.
I20260924 15:22:28.513441 19834 tablet_bootstrap.cc:654] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:28.513777 19834 log.cc:824] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:28.514073 19830 raft_consensus.cc:348] T 7e195afc8ab143228e0d3f4f0a92965f P 962f127e5e914b2da82dac5549e512bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76337a949ae4cbda4bb81d6c95c8f2e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 44059 } } peers { permanent_uuid: "962f127e5e914b2da82dac5549e512bc" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 41025 } } peers { permanent_uuid: "426e2483c5a84543ab7c4feb99460578" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38927 } }
I20260924 15:22:28.514245 19830 raft_consensus.cc:374] T 7e195afc8ab143228e0d3f4f0a92965f P 962f127e5e914b2da82dac5549e512bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:28.514290 19830 raft_consensus.cc:729] T 7e195afc8ab143228e0d3f4f0a92965f P 962f127e5e914b2da82dac5549e512bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 962f127e5e914b2da82dac5549e512bc, State: Initialized, Role: FOLLOWER
I20260924 15:22:28.514405 19830 consensus_queue.cc:260] T 7e195afc8ab143228e0d3f4f0a92965f P 962f127e5e914b2da82dac5549e512bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76337a949ae4cbda4bb81d6c95c8f2e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 44059 } } peers { permanent_uuid: "962f127e5e914b2da82dac5549e512bc" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 41025 } } peers { permanent_uuid: "426e2483c5a84543ab7c4feb99460578" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38927 } }
I20260924 15:22:28.514549 19834 tablet_bootstrap.cc:492] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e: No bootstrap required, opened a new log
I20260924 15:22:28.514444 19831 raft_consensus.cc:348] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76337a949ae4cbda4bb81d6c95c8f2e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 44059 } } peers { permanent_uuid: "962f127e5e914b2da82dac5549e512bc" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 41025 } } peers { permanent_uuid: "426e2483c5a84543ab7c4feb99460578" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38927 } }
I20260924 15:22:28.514601 19831 raft_consensus.cc:374] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:28.514609 19834 ts_tablet_manager.cc:1397] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 15:22:28.514647 19831 raft_consensus.cc:729] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 426e2483c5a84543ab7c4feb99460578, State: Initialized, Role: FOLLOWER
I20260924 15:22:28.514683 19830 ts_tablet_manager.cc:1428] T 7e195afc8ab143228e0d3f4f0a92965f P 962f127e5e914b2da82dac5549e512bc: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 15:22:28.514770 19831 consensus_queue.cc:260] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76337a949ae4cbda4bb81d6c95c8f2e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 44059 } } peers { permanent_uuid: "962f127e5e914b2da82dac5549e512bc" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 41025 } } peers { permanent_uuid: "426e2483c5a84543ab7c4feb99460578" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38927 } }
I20260924 15:22:28.514813 19549 heartbeater.cc:503] Master 127.31.12.126:36343 was elected leader, sending a full tablet report...
I20260924 15:22:28.515010 19831 ts_tablet_manager.cc:1428] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 15:22:28.515046 19816 heartbeater.cc:503] Master 127.31.12.126:36343 was elected leader, sending a full tablet report...
I20260924 15:22:28.517146 19834 raft_consensus.cc:348] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76337a949ae4cbda4bb81d6c95c8f2e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 44059 } } peers { permanent_uuid: "962f127e5e914b2da82dac5549e512bc" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 41025 } } peers { permanent_uuid: "426e2483c5a84543ab7c4feb99460578" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38927 } }
I20260924 15:22:28.517258 19834 raft_consensus.cc:374] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:28.517278 19834 raft_consensus.cc:729] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76337a949ae4cbda4bb81d6c95c8f2e, State: Initialized, Role: FOLLOWER
I20260924 15:22:28.517405 19834 consensus_queue.cc:260] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76337a949ae4cbda4bb81d6c95c8f2e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 44059 } } peers { permanent_uuid: "962f127e5e914b2da82dac5549e512bc" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 41025 } } peers { permanent_uuid: "426e2483c5a84543ab7c4feb99460578" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38927 } }
I20260924 15:22:28.517702 19834 ts_tablet_manager.cc:1428] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 15:22:28.517741 19683 heartbeater.cc:503] Master 127.31.12.126:36343 was elected leader, sending a full tablet report...
W20260924 15:22:28.619346 19684 tablet.cc:2369] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:22:28.734381 19817 tablet.cc:2369] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 15:22:28.751745 19550 tablet.cc:2369] T 7e195afc8ab143228e0d3f4f0a92965f P 962f127e5e914b2da82dac5549e512bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 15:22:28.950923 19838 raft_consensus.cc:482] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 15:22:28.951040 19838 raft_consensus.cc:504] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76337a949ae4cbda4bb81d6c95c8f2e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 44059 } } peers { permanent_uuid: "962f127e5e914b2da82dac5549e512bc" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 41025 } } peers { permanent_uuid: "426e2483c5a84543ab7c4feb99460578" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38927 } }
I20260924 15:22:28.951470 19838 leader_election.cc:288] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 962f127e5e914b2da82dac5549e512bc (127.31.12.65:41025), 426e2483c5a84543ab7c4feb99460578 (127.31.12.67:38927)
I20260924 15:22:28.955474 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:28.951751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:57273 (local address 127.31.12.67:38927)
0924 15:22:28.951801 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:28.951803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:28.951923 (+ 120us) server_negotiation.cc:407] Connection header received
0924 15:22:28.951964 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:28.951965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:28.951975 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:28.952039 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:28.952257 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.952987 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.954609 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.955116 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.955186 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:28.955193 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:28.955371 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 15:22:28.955410 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 15:22:28.955474 19829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:28.951773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.31.12.66:51019 (local address 127.31.12.65:41025)
0924 15:22:28.951814 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:28.951816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:28.951951 (+ 135us) server_negotiation.cc:407] Connection header received
0924 15:22:28.951979 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:28.951980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:28.951992 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:28.952041 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 15:22:28.952266 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.952980 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.954609 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.955115 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.955186 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:28.955193 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:28.955381 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 15:22:28.955416 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:22:28.955500 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:28.951661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.67:38927 (local address 127.31.12.66:57273)
0924 15:22:28.951817 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:28.951829 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:28.951909 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:28.952059 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:28.952060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:28.952062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:28.952158 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:28.952160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.953070 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.953071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:28.954519 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:28.954523 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.955205 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.955207 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:28.955284 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:28.955290 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 15:22:28.955347 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 15:22:28.955389 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":88,"thread_start_us":70,"threads_started":1}
I20260924 15:22:28.955502 19840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:28.951661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:41025 (local address 127.31.12.66:51019)
0924 15:22:28.951817 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:28.951829 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:28.951898 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:28.952059 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:28.952060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:28.952064 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 15:22:28.952165 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:28.952167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.953070 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.953071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:28.954519 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:28.954523 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:28.955207 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:28.955207 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:28.955284 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:28.955292 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 15:22:28.955360 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 15:22:28.955389 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":120,"thread_start_us":76,"threads_started":1}
I20260924 15:22:28.955837 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e195afc8ab143228e0d3f4f0a92965f" candidate_uuid: "f76337a949ae4cbda4bb81d6c95c8f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "426e2483c5a84543ab7c4feb99460578" is_pre_election: true
I20260924 15:22:28.956007 19769 raft_consensus.cc:2454] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f76337a949ae4cbda4bb81d6c95c8f2e in term 0.
I20260924 15:22:28.956019 19501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e195afc8ab143228e0d3f4f0a92965f" candidate_uuid: "f76337a949ae4cbda4bb81d6c95c8f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962f127e5e914b2da82dac5549e512bc" is_pre_election: true
I20260924 15:22:28.956152 19501 raft_consensus.cc:2454] T 7e195afc8ab143228e0d3f4f0a92965f P 962f127e5e914b2da82dac5549e512bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f76337a949ae4cbda4bb81d6c95c8f2e in term 0.
I20260924 15:22:28.956240 19568 leader_election.cc:302] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 426e2483c5a84543ab7c4feb99460578, f76337a949ae4cbda4bb81d6c95c8f2e; no voters:
I20260924 15:22:28.956357 19838 raft_consensus.cc:2781] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 15:22:28.956447 19838 raft_consensus.cc:482] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 15:22:28.956478 19838 raft_consensus.cc:3037] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:28.957192 19838 raft_consensus.cc:504] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76337a949ae4cbda4bb81d6c95c8f2e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 44059 } } peers { permanent_uuid: "962f127e5e914b2da82dac5549e512bc" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 41025 } } peers { permanent_uuid: "426e2483c5a84543ab7c4feb99460578" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38927 } }
I20260924 15:22:28.957326 19838 leader_election.cc:288] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [CANDIDATE]: Term 1 election: Requested vote from peers 962f127e5e914b2da82dac5549e512bc (127.31.12.65:41025), 426e2483c5a84543ab7c4feb99460578 (127.31.12.67:38927)
I20260924 15:22:28.957541 19769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e195afc8ab143228e0d3f4f0a92965f" candidate_uuid: "f76337a949ae4cbda4bb81d6c95c8f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "426e2483c5a84543ab7c4feb99460578"
I20260924 15:22:28.957564 19501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e195afc8ab143228e0d3f4f0a92965f" candidate_uuid: "f76337a949ae4cbda4bb81d6c95c8f2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962f127e5e914b2da82dac5549e512bc"
I20260924 15:22:28.957628 19769 raft_consensus.cc:3037] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:28.957628 19501 raft_consensus.cc:3037] T 7e195afc8ab143228e0d3f4f0a92965f P 962f127e5e914b2da82dac5549e512bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:28.958492 19501 raft_consensus.cc:2454] T 7e195afc8ab143228e0d3f4f0a92965f P 962f127e5e914b2da82dac5549e512bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f76337a949ae4cbda4bb81d6c95c8f2e in term 1.
I20260924 15:22:28.958492 19769 raft_consensus.cc:2454] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f76337a949ae4cbda4bb81d6c95c8f2e in term 1.
I20260924 15:22:28.958645 19568 leader_election.cc:302] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 426e2483c5a84543ab7c4feb99460578, f76337a949ae4cbda4bb81d6c95c8f2e; no voters:
I20260924 15:22:28.958729 19838 raft_consensus.cc:2781] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:28.958873 19838 raft_consensus.cc:686] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [term 1 LEADER]: Becoming Leader. State: Replica: f76337a949ae4cbda4bb81d6c95c8f2e, State: Running, Role: LEADER
I20260924 15:22:28.958981 19838 consensus_queue.cc:237] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76337a949ae4cbda4bb81d6c95c8f2e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 44059 } } peers { permanent_uuid: "962f127e5e914b2da82dac5549e512bc" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 41025 } } peers { permanent_uuid: "426e2483c5a84543ab7c4feb99460578" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38927 } }
I20260924 15:22:28.959745 19357 catalog_manager.cc:5495] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e reported cstate change: term changed from 0 to 1, leader changed from <none> to f76337a949ae4cbda4bb81d6c95c8f2e (127.31.12.66). New cstate: current_term: 1 leader_uuid: "f76337a949ae4cbda4bb81d6c95c8f2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f76337a949ae4cbda4bb81d6c95c8f2e" member_type: VOTER last_known_addr { host: "127.31.12.66" port: 44059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "962f127e5e914b2da82dac5549e512bc" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 41025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "426e2483c5a84543ab7c4feb99460578" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38927 } health_report { overall_health: UNKNOWN } } }
I20260924 15:22:29.052963 19845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:29.050540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59950 (local address 127.31.12.66:44059)
0924 15:22:29.050655 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:29.050657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:29.050665 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:22:29.050702 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:29.050703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:29.050722 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:29.050796 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:22:29.050962 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:29.051441 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:29.052203 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:29.052419 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:29.052477 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:29.052613 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:22:29.052718 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:22:29.052780 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:29.052874 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 15:22:29.052897 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":63,"threads_started":1}
I20260924 15:22:29.054986 19501 raft_consensus.cc:1261] T 7e195afc8ab143228e0d3f4f0a92965f P 962f127e5e914b2da82dac5549e512bc [term 1 FOLLOWER]: Refusing update from remote peer f76337a949ae4cbda4bb81d6c95c8f2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:22:29.054981 19769 raft_consensus.cc:1261] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578 [term 1 FOLLOWER]: Refusing update from remote peer f76337a949ae4cbda4bb81d6c95c8f2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:22:29.055395 19838 consensus_queue.cc:1035] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "962f127e5e914b2da82dac5549e512bc" member_type: VOTER last_known_addr { host: "127.31.12.65" port: 41025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:22:29.055567 19841 consensus_queue.cc:1035] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "426e2483c5a84543ab7c4feb99460578" member_type: VOTER last_known_addr { host: "127.31.12.67" port: 38927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:22:29.057850 19846 mvcc.cc:205] Tried to move back new op lower bound from 7332918677726457856 to 7332918677336522752. Current Snapshot: MvccSnapshot[applied={T|T < 7332918677726457856}]
I20260924 15:22:29.058130 19850 mvcc.cc:205] Tried to move back new op lower bound from 7332918677726457856 to 7332918677336522752. Current Snapshot: MvccSnapshot[applied={T|T < 7332918677726457856}]
I20260924 15:22:29.058610 19849 mvcc.cc:205] Tried to move back new op lower bound from 7332918677726457856 to 7332918677336522752. Current Snapshot: MvccSnapshot[applied={T|T < 7332918677726457856}]
I20260924 15:22:29.060519 19358 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:22:29.060607 19358 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 15:22:29.061348 19358 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 44284241f96a4fb0a0ac43891f649cae: Sending DeleteTablet for 3 replicas of tablet 7e195afc8ab143228e0d3f4f0a92965f
I20260924 15:22:29.061671 19615 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e195afc8ab143228e0d3f4f0a92965f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:22:29 UTC) from {username='slave'} at 127.0.0.1:59940
I20260924 15:22:29.061668 19749 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e195afc8ab143228e0d3f4f0a92965f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:22:29 UTC) from {username='slave'} at 127.0.0.1:35264
I20260924 15:22:29.061888 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 19419
I20260924 15:22:29.061924 19481 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e195afc8ab143228e0d3f4f0a92965f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 15:22:29 UTC) from {username='slave'} at 127.0.0.1:57892
I20260924 15:22:29.062573 19858 tablet_replica.cc:330] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e: stopping tablet replica
I20260924 15:22:29.062706 19858 raft_consensus.cc:2229] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [term 1 LEADER]: Raft consensus shutting down.
I20260924 15:22:29.062829 19858 raft_consensus.cc:2258] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:22:29.063169 19858 ts_tablet_manager.cc:1905] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:22:29.065289 19858 ts_tablet_manager.cc:1918] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:22:29.065374 19858 log.cc:1196] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-1/wal/wals/7e195afc8ab143228e0d3f4f0a92965f
I20260924 15:22:29.065712 19858 ts_tablet_manager.cc:1939] T 7e195afc8ab143228e0d3f4f0a92965f P f76337a949ae4cbda4bb81d6c95c8f2e: Deleting consensus metadata
I20260924 15:22:29.066089 19857 tablet_replica.cc:330] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578: stopping tablet replica
I20260924 15:22:29.066165 19345 catalog_manager.cc:4841] TS f76337a949ae4cbda4bb81d6c95c8f2e (127.31.12.66:44059): tablet 7e195afc8ab143228e0d3f4f0a92965f (table test-table [id=ac0d6587cd0e4eb1b895dd7073acb1bc]) successfully deleted
I20260924 15:22:29.066250 19857 raft_consensus.cc:2229] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 15:22:29.066375 19857 raft_consensus.cc:2258] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 15:22:29.066730 19857 ts_tablet_manager.cc:1905] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 15:22:29.068652 19857 ts_tablet_manager.cc:1918] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 15:22:29.068773 19857 log.cc:1196] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578: Deleting WAL directory at /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790263080303576-31793-0/minicluster-data/ts-2/wal/wals/7e195afc8ab143228e0d3f4f0a92965f
I20260924 15:22:29.069082 19857 ts_tablet_manager.cc:1939] T 7e195afc8ab143228e0d3f4f0a92965f P 426e2483c5a84543ab7c4feb99460578: Deleting consensus metadata
I20260924 15:22:29.069527 19343 catalog_manager.cc:4841] TS 426e2483c5a84543ab7c4feb99460578 (127.31.12.67:38927): tablet 7e195afc8ab143228e0d3f4f0a92965f (table test-table [id=ac0d6587cd0e4eb1b895dd7073acb1bc]) successfully deleted
W20260924 15:22:29.070163 19345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.31.12.65:41025 (error 108)
W20260924 15:22:29.070690 19860 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:29.070488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.65:41025 (local address 127.0.0.1:57900)
0924 15:22:29.070582 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:29.070633 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.65:41025: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":35,"threads_started":1}
W20260924 15:22:29.070811 19345 catalog_manager.cc:4584] TS 962f127e5e914b2da82dac5549e512bc (127.31.12.65:41025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e195afc8ab143228e0d3f4f0a92965f: Network error: Client connection negotiation failed: client connection to 127.31.12.65:41025: connect: Connection reset by peer (error 104)
I20260924 15:22:29.071054 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 19552
I20260924 15:22:29.077239 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 19686
I20260924 15:22:29.081430 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 19327
2026-09-24T15:22:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1132 ms)
[----------] 6 tests from EncryptionPolicyTest (8906 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:29 dist-test-slave-zwbb krb5kdc[19863](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:29 dist-test-slave-zwbb krb5kdc[19863](info): set up 2 sockets
Sep 24 15:22:29 dist-test-slave-zwbb krb5kdc[19863](info): commencing operation
krb5kdc: starting...
W20260924 15:22:30.107066 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:30 dist-test-slave-zwbb krb5kdc[19863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:22:30.222039 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43525
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:30.300635 19877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:30.300822 19877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:30.300851 19877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:30.301936 19877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:30.301970 19877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:30.301985 19877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:30.301998 19877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:30.302011 19877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:30.303357 19877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43525
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:30.303532 19877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:30.303684 19877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:30.308287 19883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:30.308301 19882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:30.308296 19885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:30.308561 19877 server_base.cc:1034] running on GCE node
I20260924 15:22:30.308795 19877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:30.309092 19877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:30.310262 19877 hybrid_clock.cc:648] HybridClock initialized: now 1790263350310252 us; error 37 us; skew 500 ppm
Sep 24 15:22:30 dist-test-slave-zwbb krb5kdc[19863](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:30.319144 19877 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:22:30.319564 19877 webserver.cc:460] Webserver started at http://127.0.0.1:45783/ using document root <none> and password file <none>
I20260924 15:22:30.319799 19877 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:30.319864 19877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:30.320012 19877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:30.321009 19877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "99bd7ce10fd84faaa2fc7a6ee6d1b004"
format_stamp: "Formatted at 2026-09-24 15:22:30 on dist-test-slave-zwbb"
server_key: "cbc8b0728c0e0ab8dce98249d6afbd41"
server_key_iv: "60211365508ac99f505891ad15339b81"
server_key_version: "encryptionkey@0"
I20260924 15:22:30.321394 19877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "99bd7ce10fd84faaa2fc7a6ee6d1b004"
format_stamp: "Formatted at 2026-09-24 15:22:30 on dist-test-slave-zwbb"
server_key: "cbc8b0728c0e0ab8dce98249d6afbd41"
server_key_iv: "60211365508ac99f505891ad15339b81"
server_key_version: "encryptionkey@0"
I20260924 15:22:30.322707 19877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:22:30.323498 19892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:30.323683 19877 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:22:30.323788 19877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "99bd7ce10fd84faaa2fc7a6ee6d1b004"
format_stamp: "Formatted at 2026-09-24 15:22:30 on dist-test-slave-zwbb"
server_key: "cbc8b0728c0e0ab8dce98249d6afbd41"
server_key_iv: "60211365508ac99f505891ad15339b81"
server_key_version: "encryptionkey@0"
I20260924 15:22:30.323863 19877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:30.338713 19877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:30.340802 19877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:30.340956 19877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:30.344928 19877 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43525
I20260924 15:22:30.344961 19945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43525 every 8 connection(s)
I20260924 15:22:30.345279 19877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:30.346184 19946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:30.346407 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 19877
I20260924 15:22:30.346524 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:30.346704 31793 external_mini_cluster.cc:1428] Setting key e1e29a58a6242092f6c3a863fc85976b
I20260924 15:22:30.349059 19946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004: Bootstrap starting.
I20260924 15:22:30.349699 19946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:30.349982 19946 log.cc:824] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:30.350695 19946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004: No bootstrap required, opened a new log
I20260924 15:22:30.352468 19946 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99bd7ce10fd84faaa2fc7a6ee6d1b004" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43525 } }
I20260924 15:22:30.352626 19946 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:30.352663 19946 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99bd7ce10fd84faaa2fc7a6ee6d1b004, State: Initialized, Role: FOLLOWER
I20260924 15:22:30.352788 19946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99bd7ce10fd84faaa2fc7a6ee6d1b004" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43525 } }
I20260924 15:22:30.352876 19946 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:30.352908 19946 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:30.352963 19946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 15:22:30 dist-test-slave-zwbb krb5kdc[19863](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 15:22:30.353677 19946 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99bd7ce10fd84faaa2fc7a6ee6d1b004" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43525 } }
I20260924 15:22:30.353790 19946 leader_election.cc:302] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99bd7ce10fd84faaa2fc7a6ee6d1b004; no voters:
I20260924 15:22:30.353983 19946 leader_election.cc:288] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:30.354024 19951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:30.354137 19951 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004 [term 1 LEADER]: Becoming Leader. State: Replica: 99bd7ce10fd84faaa2fc7a6ee6d1b004, State: Running, Role: LEADER
I20260924 15:22:30.354215 19951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99bd7ce10fd84faaa2fc7a6ee6d1b004" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43525 } }
I20260924 15:22:30.354246 19946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:30.354606 19953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99bd7ce10fd84faaa2fc7a6ee6d1b004. Latest consensus state: current_term: 1 leader_uuid: "99bd7ce10fd84faaa2fc7a6ee6d1b004" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99bd7ce10fd84faaa2fc7a6ee6d1b004" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43525 } } }
I20260924 15:22:30.354691 19953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:30.355754 19952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99bd7ce10fd84faaa2fc7a6ee6d1b004" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99bd7ce10fd84faaa2fc7a6ee6d1b004" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43525 } } }
I20260924 15:22:30.356187 19952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004 [sys.catalog]: This master's current role is: LEADER
W20260924 15:22:30.356070 19967 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:22:30.356662 19967 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:22:30.356719 19957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:30.356900 19957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:30.358357 19949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:30.347219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46843 (local address 127.0.0.1:43525)
0924 15:22:30.347393 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:30.347414 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:30.347437 (+ 23us) server_negotiation.cc:407] Connection header received
0924 15:22:30.347813 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:30.347819 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:30.347859 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:30.347973 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:30.348771 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:30.349788 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:30.350697 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:30.351172 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:30.351288 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:30.353859 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:30.353863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:30.353864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:30.353875 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:30.356494 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:30.356968 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:30.356971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:30.356971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:30.357071 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:30.357362 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:30.357363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:30.357363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:30.357714 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:30.357872 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:30.358055 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 15:22:30.358159 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":80,"threads_started":1}
I20260924 15:22:30.358883 19957 catalog_manager.cc:1347] Generated new cluster ID: 756d14b603384600bacb13510d66fdbd
I20260924 15:22:30.359062 19957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:30.359366 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:22:30.366545 19957 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:30.367211 19957 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:30.371876 19949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:30.365771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39379 (local address 127.0.0.1:43525)
0924 15:22:30.365807 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:30.365809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:30.365959 (+ 150us) server_negotiation.cc:407] Connection header received
0924 15:22:30.366030 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:30.366031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:30.366040 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:30.366096 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:30.366748 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:30.367222 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:30.367936 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:30.368151 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:30.368225 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:30.369390 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:30.369391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:30.369392 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:30.369398 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:30.370757 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:30.371120 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:30.371121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:30.371122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:30.371177 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:30.371394 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:30.371395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:30.371395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:30.371497 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:30.371606 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:30.371725 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 15:22:30.371789 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:22:30.373054 19957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99bd7ce10fd84faaa2fc7a6ee6d1b004: Generated new TSK 0
I20260924 15:22:30.373200 19957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:30.378340 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 19877
W20260924 15:22:30.382678 19861 connection.cc:535] client connection to 127.0.0.1:43525 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43525): Connection reset by peer (error 104)
2026-09-24T15:22:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:30 dist-test-slave-zwbb krb5kdc[19977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:30 dist-test-slave-zwbb krb5kdc[19977](info): set up 2 sockets
Sep 24 15:22:30 dist-test-slave-zwbb krb5kdc[19977](info): commencing operation
krb5kdc: starting...
W20260924 15:22:31.418641 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:31 dist-test-slave-zwbb krb5kdc[19977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:22:31.530009 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40867
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:31.608693 19990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:31.608883 19990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:31.608906 19990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:31.610080 19990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:31.610121 19990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:31.610133 19990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:31.610188 19990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:31.610213 19990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:31.611644 19990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40867
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:31.611912 19990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:31.612141 19990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:31.617036 19998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:31.617067 19996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:31.617081 19990 server_base.cc:1034] running on GCE node
W20260924 15:22:31.617049 19995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:31.617514 19990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:31.617802 19990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:31.618954 19990 hybrid_clock.cc:648] HybridClock initialized: now 1790263351618941 us; error 27 us; skew 500 ppm
Sep 24 15:22:31 dist-test-slave-zwbb krb5kdc[19977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:31.621871 19990 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:22:31.622318 19990 webserver.cc:460] Webserver started at http://127.0.0.1:39461/ using document root <none> and password file <none>
I20260924 15:22:31.622560 19990 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:31.622622 19990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:31.622746 19990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:31.623749 19990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "29b45ef188b346afbec9f9ae17173c75"
format_stamp: "Formatted at 2026-09-24 15:22:31 on dist-test-slave-zwbb"
server_key: "38a28864efaba0217093fadc6e3f0395"
server_key_iv: "01b1cb987866cee8ab8fd139d069c882"
server_key_version: "encryptionkey@0"
I20260924 15:22:31.624116 19990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "29b45ef188b346afbec9f9ae17173c75"
format_stamp: "Formatted at 2026-09-24 15:22:31 on dist-test-slave-zwbb"
server_key: "38a28864efaba0217093fadc6e3f0395"
server_key_iv: "01b1cb987866cee8ab8fd139d069c882"
server_key_version: "encryptionkey@0"
I20260924 15:22:31.625377 19990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:22:31.626104 20005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:31.626255 19990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:31.626350 19990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "29b45ef188b346afbec9f9ae17173c75"
format_stamp: "Formatted at 2026-09-24 15:22:31 on dist-test-slave-zwbb"
server_key: "38a28864efaba0217093fadc6e3f0395"
server_key_iv: "01b1cb987866cee8ab8fd139d069c882"
server_key_version: "encryptionkey@0"
I20260924 15:22:31.626452 19990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:31.638971 19990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:31.640635 19990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:31.640786 19990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:31.644801 19990 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40867
I20260924 15:22:31.644851 20058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40867 every 8 connection(s)
I20260924 15:22:31.645223 19990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:31.646102 20059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:31.648248 20059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75: Bootstrap starting.
I20260924 15:22:31.648756 20059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:31.649000 20059 log.cc:824] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:31.649646 20059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75: No bootstrap required, opened a new log
I20260924 15:22:31.651114 20059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b45ef188b346afbec9f9ae17173c75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40867 } }
I20260924 15:22:31.651326 20059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:31.651402 20059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29b45ef188b346afbec9f9ae17173c75, State: Initialized, Role: FOLLOWER
I20260924 15:22:31.651537 20059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b45ef188b346afbec9f9ae17173c75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40867 } }
I20260924 15:22:31.651597 20059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:31.651618 20059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:31.651643 20059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:31.652088 20059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b45ef188b346afbec9f9ae17173c75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40867 } }
I20260924 15:22:31.652169 20059 leader_election.cc:302] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29b45ef188b346afbec9f9ae17173c75; no voters:
I20260924 15:22:31.652278 20059 leader_election.cc:288] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:31.652318 20062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:31.652454 20062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75 [term 1 LEADER]: Becoming Leader. State: Replica: 29b45ef188b346afbec9f9ae17173c75, State: Running, Role: LEADER
I20260924 15:22:31.652522 20059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:31.652575 20062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b45ef188b346afbec9f9ae17173c75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40867 } }
I20260924 15:22:31.653041 20064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29b45ef188b346afbec9f9ae17173c75. Latest consensus state: current_term: 1 leader_uuid: "29b45ef188b346afbec9f9ae17173c75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b45ef188b346afbec9f9ae17173c75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40867 } } }
I20260924 15:22:31.653118 20064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:31.653177 20063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29b45ef188b346afbec9f9ae17173c75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29b45ef188b346afbec9f9ae17173c75" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40867 } } }
I20260924 15:22:31.653259 20063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:31.653551 20071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:31.654098 20071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:31.654212 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 19990
I20260924 15:22:31.654327 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:31.654515 31793 external_mini_cluster.cc:1428] Setting key 1288a24ec5818a0b5ab9d0f6441529bf
I20260924 15:22:31.655804 20071 catalog_manager.cc:1347] Generated new cluster ID: b9792d604754467b8a3f29397e3f9bc0
I20260924 15:22:31.655864 20071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:31.660310 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:31.655122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60421 (local address 127.0.0.1:40867)
0924 15:22:31.655304 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:31.655314 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:31.655365 (+ 51us) server_negotiation.cc:407] Connection header received
0924 15:22:31.655490 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:31.655495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:31.655535 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:31.655666 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:31.655907 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:31.657223 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:31.658342 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:31.658918 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:31.659019 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:31.659693 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:31.659696 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:31.659696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:31.659886 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:31.659921 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:31.659934 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:31.660003 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:31.660112 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 15:22:31.660135 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
I20260924 15:22:31.661248 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:22:31.666575 20071 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:31.666698 20081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:31.662942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50329 (local address 127.0.0.1:40867)
0924 15:22:31.662976 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:31.662978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:31.663122 (+ 144us) server_negotiation.cc:407] Connection header received
0924 15:22:31.663174 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:31.663174 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:31.663184 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:31.663244 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:31.663384 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:31.664094 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:31.665158 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:31.665492 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:31.665557 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:31.666398 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:31.666400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:31.666400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:31.666404 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:31.666421 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:31.666426 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:31.666485 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:31.666573 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 15:22:31.666585 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:22:31.667522 20071 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:31.678318 20071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29b45ef188b346afbec9f9ae17173c75: Generated new TSK 0
I20260924 15:22:31.678491 20071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:31.685724 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 19990
W20260924 15:22:31.689987 19975 connection.cc:535] client connection to 127.0.0.1:40867 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40867): Connection reset by peer (error 104)
2026-09-24T15:22:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:31 dist-test-slave-zwbb krb5kdc[20090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:31 dist-test-slave-zwbb krb5kdc[20090](info): set up 2 sockets
Sep 24 15:22:31 dist-test-slave-zwbb krb5kdc[20090](info): commencing operation
krb5kdc: starting...
W20260924 15:22:32.707818 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:32 dist-test-slave-zwbb krb5kdc[20090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263352, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:22:32.827917 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43021
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:32.908643 20103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:32.908843 20103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:32.908865 20103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:32.910130 20103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:32.910180 20103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:32.910194 20103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:32.910207 20103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:32.910265 20103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:32.911756 20103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43021
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:32.912009 20103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:32.912250 20103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:32.917109 20111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:32.917120 20108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:32.917120 20109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:32.917332 20103 server_base.cc:1034] running on GCE node
I20260924 15:22:32.917670 20103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:32.918020 20103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:32.919167 20103 hybrid_clock.cc:648] HybridClock initialized: now 1790263352919144 us; error 35 us; skew 500 ppm
Sep 24 15:22:32 dist-test-slave-zwbb krb5kdc[20090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263352, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:32.921669 20103 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:22:32.922060 20103 webserver.cc:460] Webserver started at http://127.0.0.1:33721/ using document root <none> and password file <none>
I20260924 15:22:32.922266 20103 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:32.922309 20103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:32.922406 20103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:32.923426 20103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "550a0928fdb54ff38da97333867781b4"
format_stamp: "Formatted at 2026-09-24 15:22:32 on dist-test-slave-zwbb"
server_key: "54b57401e5011e18bf91d7713bde4aac"
server_key_iv: "f800a07c6f4ae75e148e4dde4d074483"
server_key_version: "encryptionkey@0"
I20260924 15:22:32.923736 20103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "550a0928fdb54ff38da97333867781b4"
format_stamp: "Formatted at 2026-09-24 15:22:32 on dist-test-slave-zwbb"
server_key: "54b57401e5011e18bf91d7713bde4aac"
server_key_iv: "f800a07c6f4ae75e148e4dde4d074483"
server_key_version: "encryptionkey@0"
I20260924 15:22:32.925006 20103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:22:32.925736 20118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:32.925904 20103 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:22:32.926014 20103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "550a0928fdb54ff38da97333867781b4"
format_stamp: "Formatted at 2026-09-24 15:22:32 on dist-test-slave-zwbb"
server_key: "54b57401e5011e18bf91d7713bde4aac"
server_key_iv: "f800a07c6f4ae75e148e4dde4d074483"
server_key_version: "encryptionkey@0"
I20260924 15:22:32.926093 20103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:32.929387 20103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:32.931195 20103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:32.931362 20103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:32.935307 20103 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43021
I20260924 15:22:32.935355 20171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43021 every 8 connection(s)
I20260924 15:22:32.935701 20103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:32.936677 20172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:32.938863 20172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4: Bootstrap starting.
I20260924 15:22:32.939366 20172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:32.939618 20172 log.cc:824] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:32.940287 20172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4: No bootstrap required, opened a new log
I20260924 15:22:32.941879 20172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "550a0928fdb54ff38da97333867781b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43021 } }
I20260924 15:22:32.942018 20172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:32.942040 20172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 550a0928fdb54ff38da97333867781b4, State: Initialized, Role: FOLLOWER
I20260924 15:22:32.942137 20172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "550a0928fdb54ff38da97333867781b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43021 } }
I20260924 15:22:32.942221 20172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:32.942257 20172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:32.942327 20172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:32.942597 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 20103
I20260924 15:22:32.942710 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:32.942930 31793 external_mini_cluster.cc:1428] Setting key 7e9f5e2bcf2b343295bbfd5b11f46086
I20260924 15:22:32.943073 20172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "550a0928fdb54ff38da97333867781b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43021 } }
I20260924 15:22:32.943197 20172 leader_election.cc:302] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 550a0928fdb54ff38da97333867781b4; no voters:
I20260924 15:22:32.943353 20172 leader_election.cc:288] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:32.943397 20175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:32.943601 20175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4 [term 1 LEADER]: Becoming Leader. State: Replica: 550a0928fdb54ff38da97333867781b4, State: Running, Role: LEADER
I20260924 15:22:32.943703 20172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:32.943751 20175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "550a0928fdb54ff38da97333867781b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43021 } }
I20260924 15:22:32.944168 20178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "550a0928fdb54ff38da97333867781b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "550a0928fdb54ff38da97333867781b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43021 } } }
I20260924 15:22:32.944180 20179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 550a0928fdb54ff38da97333867781b4. Latest consensus state: current_term: 1 leader_uuid: "550a0928fdb54ff38da97333867781b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "550a0928fdb54ff38da97333867781b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43021 } } }
I20260924 15:22:32.944300 20178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:32.944373 20179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:32.944798 20181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:32.945606 20181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:32.945847 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:32.943370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47227 (local address 127.0.0.1:43021)
0924 15:22:32.943802 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:32.943811 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:32.943847 (+ 36us) server_negotiation.cc:407] Connection header received
0924 15:22:32.943982 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:32.943990 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:32.944025 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:32.944162 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:32.945165 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:32.945168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:32.945169 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:32.945174 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:32.945217 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:32.945234 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:32.945305 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:32.945545 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 15:22:32.945589 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":74,"threads_started":1}
I20260924 15:22:32.946991 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:22:32.948694 20181 catalog_manager.cc:1347] Generated new cluster ID: c0c054f7fae34d69b42caca6eff81ed3
I20260924 15:22:32.948761 20181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:32.949672 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:32.948463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40649 (local address 127.0.0.1:43021)
0924 15:22:32.948496 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:32.948498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:32.948680 (+ 182us) server_negotiation.cc:407] Connection header received
0924 15:22:32.948775 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:32.948776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:32.948782 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:32.948813 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:32.949459 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:32.949460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:32.949461 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:32.949465 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:32.949478 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:32.949481 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:32.949510 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:32.949605 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 15:22:32.949617 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 15:22:32.962618 20181 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:32.962798 20181 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:32.968844 20181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 550a0928fdb54ff38da97333867781b4: Generated new TSK 0
I20260924 15:22:32.969061 20181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:32.971786 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 20103
2026-09-24T15:22:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1286 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:33 dist-test-slave-zwbb krb5kdc[20203](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:33 dist-test-slave-zwbb krb5kdc[20203](info): set up 2 sockets
Sep 24 15:22:33 dist-test-slave-zwbb krb5kdc[20203](info): commencing operation
krb5kdc: starting...
W20260924 15:22:34.004173 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:34 dist-test-slave-zwbb krb5kdc[20203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:22:34.123582 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45759
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:34.207554 20216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:34.207789 20216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:34.207827 20216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:34.208981 20216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:34.209048 20216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:34.209074 20216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:34.209105 20216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:34.209129 20216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:34.210654 20216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:34.210939 20216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:34.211162 20216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:34.216022 20224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:34.216126 20216 server_base.cc:1034] running on GCE node
W20260924 15:22:34.216027 20221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:34.216029 20222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:34.216481 20216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:34.216724 20216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:34.217890 20216 hybrid_clock.cc:648] HybridClock initialized: now 1790263354217884 us; error 37 us; skew 500 ppm
Sep 24 15:22:34 dist-test-slave-zwbb krb5kdc[20203](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:34.220295 20216 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:22:34.220784 20216 webserver.cc:460] Webserver started at http://127.0.0.1:46049/ using document root <none> and password file <none>
I20260924 15:22:34.221026 20216 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:34.221095 20216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:34.221247 20216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:34.222350 20216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "4f3d0716b4644a3cbf4fd3fe1cc8f7a4"
format_stamp: "Formatted at 2026-09-24 15:22:34 on dist-test-slave-zwbb"
server_key: "896978f38a2c5b5c6d994870bfe99c62"
server_key_iv: "a548b662ccb538080c961ca897eb4c0a"
server_key_version: "encryptionkey@0"
I20260924 15:22:34.222729 20216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "4f3d0716b4644a3cbf4fd3fe1cc8f7a4"
format_stamp: "Formatted at 2026-09-24 15:22:34 on dist-test-slave-zwbb"
server_key: "896978f38a2c5b5c6d994870bfe99c62"
server_key_iv: "a548b662ccb538080c961ca897eb4c0a"
server_key_version: "encryptionkey@0"
I20260924 15:22:34.224073 20216 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:22:34.224800 20231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:34.224996 20216 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:34.225097 20216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "4f3d0716b4644a3cbf4fd3fe1cc8f7a4"
format_stamp: "Formatted at 2026-09-24 15:22:34 on dist-test-slave-zwbb"
server_key: "896978f38a2c5b5c6d994870bfe99c62"
server_key_iv: "a548b662ccb538080c961ca897eb4c0a"
server_key_version: "encryptionkey@0"
I20260924 15:22:34.225179 20216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:34.239866 20216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:34.241801 20216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:34.241971 20216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:34.246156 20216 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45759
I20260924 15:22:34.246214 20284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45759 every 8 connection(s)
I20260924 15:22:34.246558 20216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:34.247387 20285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:34.248386 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 20216
I20260924 15:22:34.248517 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:34.248688 31793 external_mini_cluster.cc:1428] Setting key a34352d9a006717647b3625a95c3b648
I20260924 15:22:34.249998 20285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4: Bootstrap starting.
I20260924 15:22:34.250733 20285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:34.251060 20285 log.cc:824] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:34.251793 20285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4: No bootstrap required, opened a new log
I20260924 15:22:34.254235 20285 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f3d0716b4644a3cbf4fd3fe1cc8f7a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } }
I20260924 15:22:34.254429 20285 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:34.254470 20285 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f3d0716b4644a3cbf4fd3fe1cc8f7a4, State: Initialized, Role: FOLLOWER
I20260924 15:22:34.254604 20285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f3d0716b4644a3cbf4fd3fe1cc8f7a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } }
I20260924 15:22:34.254693 20285 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:34.254727 20285 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:34.254783 20285 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:34.255563 20285 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f3d0716b4644a3cbf4fd3fe1cc8f7a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } }
I20260924 15:22:34.255654 20285 leader_election.cc:302] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f3d0716b4644a3cbf4fd3fe1cc8f7a4; no voters:
I20260924 15:22:34.255878 20285 leader_election.cc:288] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:34.255900 20290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4 [term 1 FOLLOWER]: Leader election won for term 1
Sep 24 15:22:34 dist-test-slave-zwbb krb5kdc[20203](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 15:22:34.256062 20290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4 [term 1 LEADER]: Becoming Leader. State: Replica: 4f3d0716b4644a3cbf4fd3fe1cc8f7a4, State: Running, Role: LEADER
I20260924 15:22:34.256150 20285 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:34.256132 20290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f3d0716b4644a3cbf4fd3fe1cc8f7a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } }
I20260924 15:22:34.256578 20291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f3d0716b4644a3cbf4fd3fe1cc8f7a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f3d0716b4644a3cbf4fd3fe1cc8f7a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } } }
I20260924 15:22:34.256922 20291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:34.257140 20292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f3d0716b4644a3cbf4fd3fe1cc8f7a4. Latest consensus state: current_term: 1 leader_uuid: "4f3d0716b4644a3cbf4fd3fe1cc8f7a4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f3d0716b4644a3cbf4fd3fe1cc8f7a4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45759 } } }
I20260924 15:22:34.257555 20292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:34.257175 20299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:34.258349 20299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:34.260037 20299 catalog_manager.cc:1347] Generated new cluster ID: f2642f346716479c89c87e72a3b8bece
I20260924 15:22:34.260102 20299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:34.260998 20288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:34.249198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55997 (local address 127.0.0.1:45759)
0924 15:22:34.249380 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:34.249390 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:34.249425 (+ 35us) server_negotiation.cc:407] Connection header received
0924 15:22:34.249932 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:34.249938 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:34.249985 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:34.250140 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:34.250971 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:34.252112 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:34.253182 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:34.253635 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:34.253731 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:34.256662 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:34.256666 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:34.256666 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:34.256678 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:34.259243 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:34.259668 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:34.259671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:34.259671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:34.259784 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:34.260035 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:34.260037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:34.260037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:34.260335 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:34.260496 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:34.260634 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 15:22:34.260779 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20260924 15:22:34.261862 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:22:34.269263 20288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:34.263229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52783 (local address 127.0.0.1:45759)
0924 15:22:34.263267 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:34.263269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:34.263384 (+ 115us) server_negotiation.cc:407] Connection header received
0924 15:22:34.263458 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:34.263459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:34.263469 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:34.263512 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:34.264052 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:34.264522 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:34.265237 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:34.265534 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:34.265638 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:34.266794 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:34.266795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:34.266795 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:34.266801 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:34.268157 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:34.268514 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:34.268515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:34.268516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:34.268570 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:34.268766 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:34.268767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:34.268767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:34.268876 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:34.268975 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:34.269084 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 15:22:34.269161 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:22:34.274104 20299 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:34.274787 20299 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:34.295926 20299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f3d0716b4644a3cbf4fd3fe1cc8f7a4: Generated new TSK 0
I20260924 15:22:34.296146 20299 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:34.299175 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 20216
2026-09-24T15:22:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1327 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:34 dist-test-slave-zwbb krb5kdc[20316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:34 dist-test-slave-zwbb krb5kdc[20316](info): set up 2 sockets
Sep 24 15:22:34 dist-test-slave-zwbb krb5kdc[20316](info): commencing operation
krb5kdc: starting...
W20260924 15:22:35.331210 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:35 dist-test-slave-zwbb krb5kdc[20316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:22:35.446203 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40087
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:35.527426 20330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:35.527684 20330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:35.527727 20330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:35.528961 20330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:35.529032 20330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:35.529055 20330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:35.529076 20330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:35.529107 20330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:35.530583 20330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:35.530859 20330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:35.531085 20330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:35.535792 20336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:35.535777 20335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:35.535808 20330 server_base.cc:1034] running on GCE node
W20260924 15:22:35.535777 20338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:35.536201 20330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:35.536487 20330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:35.537648 20330 hybrid_clock.cc:648] HybridClock initialized: now 1790263355537647 us; error 47 us; skew 500 ppm
Sep 24 15:22:35 dist-test-slave-zwbb krb5kdc[20316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:35.544993 20330 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:22:35.545404 20330 webserver.cc:460] Webserver started at http://127.0.0.1:43255/ using document root <none> and password file <none>
I20260924 15:22:35.545640 20330 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:35.545711 20330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:35.545862 20330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:35.546862 20330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "668621a984804a3ca8f21edbf6c70b22"
format_stamp: "Formatted at 2026-09-24 15:22:35 on dist-test-slave-zwbb"
server_key: "33e41d3337b3a4b1af1051bfd76f2b73"
server_key_iv: "4282677a99f0520d8725285b5d6e66d5"
server_key_version: "encryptionkey@0"
I20260924 15:22:35.547221 20330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "668621a984804a3ca8f21edbf6c70b22"
format_stamp: "Formatted at 2026-09-24 15:22:35 on dist-test-slave-zwbb"
server_key: "33e41d3337b3a4b1af1051bfd76f2b73"
server_key_iv: "4282677a99f0520d8725285b5d6e66d5"
server_key_version: "encryptionkey@0"
I20260924 15:22:35.548528 20330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:22:35.549194 20345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:35.549356 20330 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:22:35.549474 20330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "668621a984804a3ca8f21edbf6c70b22"
format_stamp: "Formatted at 2026-09-24 15:22:35 on dist-test-slave-zwbb"
server_key: "33e41d3337b3a4b1af1051bfd76f2b73"
server_key_iv: "4282677a99f0520d8725285b5d6e66d5"
server_key_version: "encryptionkey@0"
I20260924 15:22:35.549561 20330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:35.561702 20330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:35.563318 20330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:35.563469 20330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:35.567422 20330 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40087
I20260924 15:22:35.567453 20399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40087 every 8 connection(s)
I20260924 15:22:35.567775 20330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:35.568560 20400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:35.570624 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 20330
I20260924 15:22:35.570724 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:35.570915 31793 external_mini_cluster.cc:1428] Setting key 19ce37191d998e9b853a7b95fd450159
I20260924 15:22:35.570931 20400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22: Bootstrap starting.
I20260924 15:22:35.571655 20400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:35.571961 20400 log.cc:824] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:35.572816 20400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22: No bootstrap required, opened a new log
I20260924 15:22:35.575086 20400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668621a984804a3ca8f21edbf6c70b22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40087 } }
I20260924 15:22:35.575250 20400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:35.575284 20400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 668621a984804a3ca8f21edbf6c70b22, State: Initialized, Role: FOLLOWER
I20260924 15:22:35.575399 20400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668621a984804a3ca8f21edbf6c70b22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40087 } }
I20260924 15:22:35.575488 20400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:35.575531 20400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:35.575570 20400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:35.576124 20400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668621a984804a3ca8f21edbf6c70b22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40087 } }
I20260924 15:22:35.576236 20400 leader_election.cc:302] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 668621a984804a3ca8f21edbf6c70b22; no voters:
I20260924 15:22:35.576396 20400 leader_election.cc:288] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:35.576444 20405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:35.576599 20405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22 [term 1 LEADER]: Becoming Leader. State: Replica: 668621a984804a3ca8f21edbf6c70b22, State: Running, Role: LEADER
I20260924 15:22:35.576666 20400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:35.576776 20405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668621a984804a3ca8f21edbf6c70b22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40087 } }
I20260924 15:22:35.577225 20403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:35.571420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41565 (local address 127.0.0.1:40087)
0924 15:22:35.571593 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:35.571600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:35.571623 (+ 23us) server_negotiation.cc:407] Connection header received
0924 15:22:35.572124 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:35.572131 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:35.572177 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:35.572290 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:35.572547 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:35.574008 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:35.574800 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:35.575407 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:35.575518 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:35.576343 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:35.576347 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:35.576347 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:35.576588 (+ 241us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:35.576631 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:35.576647 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:35.576961 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:35.576992 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 15:22:35.577052 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260924 15:22:35.577247 20407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "668621a984804a3ca8f21edbf6c70b22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668621a984804a3ca8f21edbf6c70b22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40087 } } }
I20260924 15:22:35.577260 20406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 668621a984804a3ca8f21edbf6c70b22. Latest consensus state: current_term: 1 leader_uuid: "668621a984804a3ca8f21edbf6c70b22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "668621a984804a3ca8f21edbf6c70b22" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40087 } } }
I20260924 15:22:35.577504 20407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:35.577505 20406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:35.577745 20414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:35.578467 20414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:35.578552 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:22:35.580276 20414 catalog_manager.cc:1347] Generated new cluster ID: 777246f7c82a4d6d9dfda4d91abdf95d
I20260924 15:22:35.580332 20414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:35.583599 20403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:35.580062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54661 (local address 127.0.0.1:40087)
0924 15:22:35.580099 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:35.580100 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:35.580227 (+ 127us) server_negotiation.cc:407] Connection header received
0924 15:22:35.580310 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:35.580311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:35.580322 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:35.580381 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:35.580564 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:35.581286 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:35.582327 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:35.582615 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:35.582654 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:35.583348 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:35.583350 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:35.583350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:35.583354 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:35.583367 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:35.583371 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:35.583398 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:35.583503 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 15:22:35.583525 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:22:35.591780 20414 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:35.592309 20414 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:35.599001 20414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 668621a984804a3ca8f21edbf6c70b22: Generated new TSK 0
I20260924 15:22:35.599172 20414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:35.602321 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 20330
2026-09-24T15:22:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:35 dist-test-slave-zwbb krb5kdc[20431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:35 dist-test-slave-zwbb krb5kdc[20431](info): set up 2 sockets
Sep 24 15:22:35 dist-test-slave-zwbb krb5kdc[20431](info): commencing operation
krb5kdc: starting...
W20260924 15:22:36.631069 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:36 dist-test-slave-zwbb krb5kdc[20431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263356, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:22:36.748869 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42871
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:36.829170 20445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:36.829352 20445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:36.829375 20445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:36.830473 20445 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:36.830516 20445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:36.830530 20445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:36.830543 20445 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:36.830555 20445 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:36.831892 20445 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:36.832074 20445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:36.832233 20445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:36.836869 20450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:36.837033 20445 server_base.cc:1034] running on GCE node
W20260924 15:22:36.836900 20451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:36.836869 20453 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:36.837340 20445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:36.837677 20445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:36.838828 20445 hybrid_clock.cc:648] HybridClock initialized: now 1790263356838806 us; error 33 us; skew 500 ppm
Sep 24 15:22:36 dist-test-slave-zwbb krb5kdc[20431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263356, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:36.841228 20445 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:22:36.841706 20445 webserver.cc:460] Webserver started at http://127.0.0.1:37557/ using document root <none> and password file <none>
I20260924 15:22:36.841941 20445 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:36.841986 20445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:36.842147 20445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:36.843214 20445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "1ab2f03816b14a66896a302c465f3c59"
format_stamp: "Formatted at 2026-09-24 15:22:36 on dist-test-slave-zwbb"
server_key: "e41f27264f658e4a6978aefe490919ea"
server_key_iv: "0e807d1a168779abc3bf86b8ff0f9856"
server_key_version: "encryptionkey@0"
I20260924 15:22:36.843556 20445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "1ab2f03816b14a66896a302c465f3c59"
format_stamp: "Formatted at 2026-09-24 15:22:36 on dist-test-slave-zwbb"
server_key: "e41f27264f658e4a6978aefe490919ea"
server_key_iv: "0e807d1a168779abc3bf86b8ff0f9856"
server_key_version: "encryptionkey@0"
I20260924 15:22:36.844879 20445 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:22:36.845644 20460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:36.845831 20445 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:22:36.845925 20445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "1ab2f03816b14a66896a302c465f3c59"
format_stamp: "Formatted at 2026-09-24 15:22:36 on dist-test-slave-zwbb"
server_key: "e41f27264f658e4a6978aefe490919ea"
server_key_iv: "0e807d1a168779abc3bf86b8ff0f9856"
server_key_version: "encryptionkey@0"
I20260924 15:22:36.845999 20445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:36.849153 20445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:36.850901 20445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:36.851042 20445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:36.855101 20445 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42871
I20260924 15:22:36.855180 20512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42871 every 8 connection(s)
I20260924 15:22:36.855458 20445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:36.856385 20513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:36.858534 20513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59: Bootstrap starting.
I20260924 15:22:36.859081 20513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:36.859361 20513 log.cc:824] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:36.860059 20513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59: No bootstrap required, opened a new log
I20260924 15:22:36.861709 20513 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab2f03816b14a66896a302c465f3c59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42871 } }
I20260924 15:22:36.861843 20513 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:36.861866 20513 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ab2f03816b14a66896a302c465f3c59, State: Initialized, Role: FOLLOWER
I20260924 15:22:36.862047 20513 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab2f03816b14a66896a302c465f3c59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42871 } }
I20260924 15:22:36.862110 20513 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:36.862149 20513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:36.862197 20513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:36.862756 20513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab2f03816b14a66896a302c465f3c59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42871 } }
I20260924 15:22:36.862861 20513 leader_election.cc:302] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ab2f03816b14a66896a302c465f3c59; no voters:
I20260924 15:22:36.863040 20513 leader_election.cc:288] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:36.863065 20517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:36.863260 20517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59 [term 1 LEADER]: Becoming Leader. State: Replica: 1ab2f03816b14a66896a302c465f3c59, State: Running, Role: LEADER
I20260924 15:22:36.863348 20513 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:36.863569 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 20445
I20260924 15:22:36.863703 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:36.863927 31793 external_mini_cluster.cc:1428] Setting key ce350d0c654fa460435284d4632333c0
I20260924 15:22:36.863389 20517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab2f03816b14a66896a302c465f3c59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42871 } }
I20260924 15:22:36.864692 20520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ab2f03816b14a66896a302c465f3c59. Latest consensus state: current_term: 1 leader_uuid: "1ab2f03816b14a66896a302c465f3c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab2f03816b14a66896a302c465f3c59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42871 } } }
I20260924 15:22:36.864866 20518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ab2f03816b14a66896a302c465f3c59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ab2f03816b14a66896a302c465f3c59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42871 } } }
I20260924 15:22:36.864953 20520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:36.864964 20518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:36.865182 20535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:36.865710 20535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:36.866500 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:36.864608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45175 (local address 127.0.0.1:42871)
0924 15:22:36.865066 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:36.865074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:36.865110 (+ 36us) server_negotiation.cc:407] Connection header received
0924 15:22:36.865225 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:36.865233 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:36.865262 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:36.865400 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:36.866061 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:36.866065 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:36.866065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:36.866069 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:36.866119 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:36.866131 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:36.866203 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:36.866273 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 15:22:36.866312 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":56,"threads_started":1}
I20260924 15:22:36.867273 20535 catalog_manager.cc:1347] Generated new cluster ID: 4fc83f65000f4077b0e0bf17c1801080
I20260924 15:22:36.867352 20535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:36.867444 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:22:36.869801 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:36.868811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42825 (local address 127.0.0.1:42871)
0924 15:22:36.868871 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:36.868873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:36.868890 (+ 17us) server_negotiation.cc:407] Connection header received
0924 15:22:36.868948 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:36.868949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:36.868954 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:36.869009 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:36.869593 (+ 584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:36.869594 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:36.869595 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:36.869599 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:36.869611 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:36.869615 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:36.869671 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:36.869726 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 15:22:36.869740 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 15:22:36.875442 20535 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:36.875595 20535 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:36.890024 20535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ab2f03816b14a66896a302c465f3c59: Generated new TSK 0
I20260924 15:22:36.890216 20535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:36.897260 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 20445
2026-09-24T15:22:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:36 dist-test-slave-zwbb krb5kdc[20546](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:36 dist-test-slave-zwbb krb5kdc[20546](info): set up 2 sockets
Sep 24 15:22:36 dist-test-slave-zwbb krb5kdc[20546](info): commencing operation
krb5kdc: starting...
W20260924 15:22:37.922972 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:37 dist-test-slave-zwbb krb5kdc[20546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:22:38.036077 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35459
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:38.115695 20559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:38.115901 20559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:38.115926 20559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:38.117151 20559 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:38.117197 20559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:38.117210 20559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:38.117223 20559 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:38.117234 20559 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:38.118763 20559 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:38.119011 20559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:38.119254 20559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:38.124308 20568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:38.124284 20565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:38.124264 20566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:38.124595 20559 server_base.cc:1034] running on GCE node
I20260924 15:22:38.124878 20559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:38.125162 20559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:38.126314 20559 hybrid_clock.cc:648] HybridClock initialized: now 1790263358126302 us; error 30 us; skew 500 ppm
Sep 24 15:22:38 dist-test-slave-zwbb krb5kdc[20546](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:38.128971 20559 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:22:38.129477 20559 webserver.cc:460] Webserver started at http://127.0.0.1:36215/ using document root <none> and password file <none>
I20260924 15:22:38.129727 20559 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:38.129771 20559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:38.129930 20559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:38.130970 20559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "6671795ff11c4fec91154a8048cbe378"
format_stamp: "Formatted at 2026-09-24 15:22:38 on dist-test-slave-zwbb"
server_key: "a0b0fbd8072e9a88fe8aa03aa4ada682"
server_key_iv: "718922c338e4aff80b51420980d972bf"
server_key_version: "encryptionkey@0"
I20260924 15:22:38.131328 20559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "6671795ff11c4fec91154a8048cbe378"
format_stamp: "Formatted at 2026-09-24 15:22:38 on dist-test-slave-zwbb"
server_key: "a0b0fbd8072e9a88fe8aa03aa4ada682"
server_key_iv: "718922c338e4aff80b51420980d972bf"
server_key_version: "encryptionkey@0"
I20260924 15:22:38.132664 20559 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:22:38.133497 20575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:38.133670 20559 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:22:38.133764 20559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "6671795ff11c4fec91154a8048cbe378"
format_stamp: "Formatted at 2026-09-24 15:22:38 on dist-test-slave-zwbb"
server_key: "a0b0fbd8072e9a88fe8aa03aa4ada682"
server_key_iv: "718922c338e4aff80b51420980d972bf"
server_key_version: "encryptionkey@0"
I20260924 15:22:38.133857 20559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:38.150768 20559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:38.152943 20559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:38.153092 20559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:38.157202 20559 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35459
I20260924 15:22:38.157248 20627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35459 every 8 connection(s)
I20260924 15:22:38.157606 20559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:38.158406 20628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:38.160506 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 20559
I20260924 15:22:38.160633 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:38.160626 20628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378: Bootstrap starting.
I20260924 15:22:38.160815 31793 external_mini_cluster.cc:1428] Setting key 8a9ad1f22d04b0a2d4a08a108e878ca8
I20260924 15:22:38.161247 20628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:38.161638 20628 log.cc:824] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:38.162375 20628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378: No bootstrap required, opened a new log
I20260924 15:22:38.164499 20628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6671795ff11c4fec91154a8048cbe378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35459 } }
I20260924 15:22:38.164667 20628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:38.164698 20628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6671795ff11c4fec91154a8048cbe378, State: Initialized, Role: FOLLOWER
I20260924 15:22:38.164808 20628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6671795ff11c4fec91154a8048cbe378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35459 } }
I20260924 15:22:38.164896 20628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:38.164934 20628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:38.164986 20628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:38.165695 20628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6671795ff11c4fec91154a8048cbe378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35459 } }
I20260924 15:22:38.165810 20628 leader_election.cc:302] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6671795ff11c4fec91154a8048cbe378; no voters:
I20260924 15:22:38.165967 20628 leader_election.cc:288] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:38.166014 20633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:38.166215 20633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378 [term 1 LEADER]: Becoming Leader. State: Replica: 6671795ff11c4fec91154a8048cbe378, State: Running, Role: LEADER
I20260924 15:22:38.166240 20628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:38.166363 20633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6671795ff11c4fec91154a8048cbe378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35459 } }
I20260924 15:22:38.166720 20634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6671795ff11c4fec91154a8048cbe378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6671795ff11c4fec91154a8048cbe378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35459 } } }
I20260924 15:22:38.166800 20634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:38.167659 20640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:38.167698 20635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6671795ff11c4fec91154a8048cbe378. Latest consensus state: current_term: 1 leader_uuid: "6671795ff11c4fec91154a8048cbe378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6671795ff11c4fec91154a8048cbe378" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35459 } } }
I20260924 15:22:38.167778 20635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378 [sys.catalog]: This master's current role is: LEADER
Sep 24 15:22:38 dist-test-slave-zwbb krb5kdc[20546](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 15:22:38.168192 20640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:38.169953 20640 catalog_manager.cc:1347] Generated new cluster ID: ac55d907b78d476da4725b36baf6340c
I20260924 15:22:38.170004 20640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:38.172108 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:38.161326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:51729 (local address 127.0.0.1:35459)
0924 15:22:38.161553 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:38.161562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:38.161594 (+ 32us) server_negotiation.cc:407] Connection header received
0924 15:22:38.162057 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:38.162064 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:38.162111 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:38.162221 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:38.162972 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:38.163815 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:38.164961 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:38.165573 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:38.165661 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:38.168541 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:38.168545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:38.168545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:38.168556 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:38.170489 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:38.170914 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:38.170916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:38.170917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:38.171016 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:38.171261 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:38.171262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:38.171262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:38.171525 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:38.171672 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:38.171808 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 15:22:38.171900 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":139,"threads_started":1}
I20260924 15:22:38.172838 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:22:38.184660 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:38.177705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:36093 (local address 127.0.0.1:35459)
0924 15:22:38.177735 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:38.177736 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:38.177850 (+ 114us) server_negotiation.cc:407] Connection header received
0924 15:22:38.177886 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:38.177886 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:38.177895 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:38.177942 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:38.178458 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:38.179240 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:38.180203 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:38.180523 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:38.180610 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:38.181920 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:38.181922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:38.181922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:38.181929 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:38.183464 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:38.183851 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:38.183852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:38.183852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:38.183909 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:38.184132 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:38.184133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:38.184133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:38.184275 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:38.184377 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:38.184507 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 15:22:38.184570 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 15:22:38.187569 20640 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:38.188006 20640 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:38.195572 20640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6671795ff11c4fec91154a8048cbe378: Generated new TSK 0
I20260924 15:22:38.195751 20640 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:38.199215 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 20559
W20260924 15:22:38.203550 20544 connection.cc:535] client connection to 127.0.0.1:35459 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35459): Connection reset by peer (error 104)
2026-09-24T15:22:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1301 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:38 dist-test-slave-zwbb krb5kdc[20660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:38 dist-test-slave-zwbb krb5kdc[20660](info): set up 2 sockets
Sep 24 15:22:38 dist-test-slave-zwbb krb5kdc[20660](info): commencing operation
krb5kdc: starting...
W20260924 15:22:39.223254 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:39 dist-test-slave-zwbb krb5kdc[20660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:22:39.338451 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43043
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:39.416515 20674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:39.416738 20674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:39.416791 20674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:39.417990 20674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:39.418053 20674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:39.418077 20674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:39.418100 20674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:39.418126 20674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:39.419550 20674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:39.419806 20674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:39.420033 20674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:39.424612 20679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:39.424618 20680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:39.424682 20682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:39.424883 20674 server_base.cc:1034] running on GCE node
I20260924 15:22:39.425215 20674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:39.425558 20674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:39.426733 20674 hybrid_clock.cc:648] HybridClock initialized: now 1790263359426710 us; error 47 us; skew 500 ppm
Sep 24 15:22:39 dist-test-slave-zwbb krb5kdc[20660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:39.429245 20674 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:22:39.429740 20674 webserver.cc:460] Webserver started at http://127.0.0.1:45673/ using document root <none> and password file <none>
I20260924 15:22:39.429984 20674 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:39.430055 20674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:39.430208 20674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:39.431214 20674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "c9bd5c7573cc444b87ee26c379c202eb"
format_stamp: "Formatted at 2026-09-24 15:22:39 on dist-test-slave-zwbb"
server_key: "f76c42de3423840937ff347a3a4ddee0"
server_key_iv: "97727f3ea50b5f4bf64ea256332ab4b6"
server_key_version: "encryptionkey@0"
I20260924 15:22:39.431571 20674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "c9bd5c7573cc444b87ee26c379c202eb"
format_stamp: "Formatted at 2026-09-24 15:22:39 on dist-test-slave-zwbb"
server_key: "f76c42de3423840937ff347a3a4ddee0"
server_key_iv: "97727f3ea50b5f4bf64ea256332ab4b6"
server_key_version: "encryptionkey@0"
I20260924 15:22:39.432780 20674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 15:22:39.433432 20689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:39.433626 20674 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:39.433736 20674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "c9bd5c7573cc444b87ee26c379c202eb"
format_stamp: "Formatted at 2026-09-24 15:22:39 on dist-test-slave-zwbb"
server_key: "f76c42de3423840937ff347a3a4ddee0"
server_key_iv: "97727f3ea50b5f4bf64ea256332ab4b6"
server_key_version: "encryptionkey@0"
I20260924 15:22:39.433821 20674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:39.452795 20674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:39.455080 20674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:39.455242 20674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:39.459437 20674 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43043
I20260924 15:22:39.459466 20741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43043 every 8 connection(s)
I20260924 15:22:39.459797 20674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:39.460563 20742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:39.462602 20742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb: Bootstrap starting.
I20260924 15:22:39.462869 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 20674
I20260924 15:22:39.462999 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:39.463197 31793 external_mini_cluster.cc:1428] Setting key dd4668f41e09ae231dd51e501067f4ca
I20260924 15:22:39.463253 20742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:39.463554 20742 log.cc:824] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:39.464278 20742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb: No bootstrap required, opened a new log
I20260924 15:22:39.466691 20742 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9bd5c7573cc444b87ee26c379c202eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43043 } }
I20260924 15:22:39.466811 20742 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:39.466833 20742 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9bd5c7573cc444b87ee26c379c202eb, State: Initialized, Role: FOLLOWER
I20260924 15:22:39.466956 20742 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9bd5c7573cc444b87ee26c379c202eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43043 } }
I20260924 15:22:39.467025 20742 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:39.467047 20742 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:39.467111 20742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:39.467648 20742 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9bd5c7573cc444b87ee26c379c202eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43043 } }
I20260924 15:22:39.467767 20742 leader_election.cc:302] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9bd5c7573cc444b87ee26c379c202eb; no voters:
I20260924 15:22:39.467924 20742 leader_election.cc:288] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:39.467959 20747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:39.468142 20747 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb [term 1 LEADER]: Becoming Leader. State: Replica: c9bd5c7573cc444b87ee26c379c202eb, State: Running, Role: LEADER
I20260924 15:22:39.468242 20747 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9bd5c7573cc444b87ee26c379c202eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43043 } }
I20260924 15:22:39.468299 20742 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:39.468632 20749 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9bd5c7573cc444b87ee26c379c202eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9bd5c7573cc444b87ee26c379c202eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43043 } } }
I20260924 15:22:39.468858 20749 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:39.468781 20748 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9bd5c7573cc444b87ee26c379c202eb. Latest consensus state: current_term: 1 leader_uuid: "c9bd5c7573cc444b87ee26c379c202eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9bd5c7573cc444b87ee26c379c202eb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43043 } } }
I20260924 15:22:39.469158 20748 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:39.469163 20756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:39.469786 20756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:39.470181 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:39.463670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:37893 (local address 127.0.0.1:43043)
0924 15:22:39.463831 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:39.463839 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:39.463868 (+ 29us) server_negotiation.cc:407] Connection header received
0924 15:22:39.464467 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:39.464476 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:39.464537 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:39.464682 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:39.464891 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:39.466328 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:39.467221 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:39.467654 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:39.467778 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:39.469168 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:39.469172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:39.469172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:39.469421 (+ 249us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:39.469481 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:39.469499 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:39.469587 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:39.469751 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 15:22:39.469781 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260924 15:22:39.471347 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:22:39.471549 20756 catalog_manager.cc:1347] Generated new cluster ID: 1c6fc37e5938412f8d5d444381f48e94
I20260924 15:22:39.471585 20756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:39.476472 20746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:39.472813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:53501 (local address 127.0.0.1:43043)
0924 15:22:39.472850 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:39.472851 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:39.473023 (+ 172us) server_negotiation.cc:407] Connection header received
0924 15:22:39.473072 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:39.473073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:39.473089 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:39.473143 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:39.473265 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:39.474073 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:39.475073 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:39.475388 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:39.475475 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:39.476156 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:39.476158 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:39.476158 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:39.476164 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:39.476181 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:39.476185 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:39.476248 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:39.476350 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 15:22:39.476366 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:22:39.499691 20756 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:39.500325 20756 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:39.506728 20756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9bd5c7573cc444b87ee26c379c202eb: Generated new TSK 0
I20260924 15:22:39.506920 20756 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:39.513799 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 20674
W20260924 15:22:39.518834 20657 connection.cc:535] client connection to 127.0.0.1:43043 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43043): Connection reset by peer (error 104)
2026-09-24T15:22:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:39 dist-test-slave-zwbb krb5kdc[20773](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:39 dist-test-slave-zwbb krb5kdc[20773](info): set up 2 sockets
Sep 24 15:22:39 dist-test-slave-zwbb krb5kdc[20773](info): commencing operation
krb5kdc: starting...
W20260924 15:22:40.579262 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:40 dist-test-slave-zwbb krb5kdc[20773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263360, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:22:40.694377 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40129
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:40.775264 20786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:40.775485 20786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:40.775523 20786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:40.776680 20786 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:40.776748 20786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:40.776772 20786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:40.776805 20786 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:40.776835 20786 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:40.778352 20786 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:40.778617 20786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:40.778831 20786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:40.783737 20792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:40.783728 20791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:40.783840 20786 server_base.cc:1034] running on GCE node
W20260924 15:22:40.783730 20794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:40.784133 20786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:40.784370 20786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:40.785514 20786 hybrid_clock.cc:648] HybridClock initialized: now 1790263360785484 us; error 50 us; skew 500 ppm
Sep 24 15:22:40 dist-test-slave-zwbb krb5kdc[20773](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263360, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:40.791252 20786 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:22:40.791734 20786 webserver.cc:460] Webserver started at http://127.0.0.1:35791/ using document root <none> and password file <none>
I20260924 15:22:40.791961 20786 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:40.792029 20786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:40.792179 20786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:40.793208 20786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "943ab0911221424e8e8bc50354a7f5d6"
format_stamp: "Formatted at 2026-09-24 15:22:40 on dist-test-slave-zwbb"
server_key: "1fb1fec443a1e46afe8d92922289d048"
server_key_iv: "d1b019a7179cc8a0996b84c872630c53"
server_key_version: "encryptionkey@0"
I20260924 15:22:40.793605 20786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "943ab0911221424e8e8bc50354a7f5d6"
format_stamp: "Formatted at 2026-09-24 15:22:40 on dist-test-slave-zwbb"
server_key: "1fb1fec443a1e46afe8d92922289d048"
server_key_iv: "d1b019a7179cc8a0996b84c872630c53"
server_key_version: "encryptionkey@0"
I20260924 15:22:40.794926 20786 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:22:40.795631 20801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:40.795818 20786 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:22:40.795917 20786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "943ab0911221424e8e8bc50354a7f5d6"
format_stamp: "Formatted at 2026-09-24 15:22:40 on dist-test-slave-zwbb"
server_key: "1fb1fec443a1e46afe8d92922289d048"
server_key_iv: "d1b019a7179cc8a0996b84c872630c53"
server_key_version: "encryptionkey@0"
I20260924 15:22:40.796000 20786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:40.799345 20786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:40.801079 20786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:40.801230 20786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:40.805243 20786 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40129
I20260924 15:22:40.805270 20855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40129 every 8 connection(s)
I20260924 15:22:40.805658 20786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:40.806571 20856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:40.808298 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 20786
I20260924 15:22:40.808416 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:40.808621 31793 external_mini_cluster.cc:1428] Setting key 359bd4ee698bce40d4a7b8b808a3fa62
I20260924 15:22:40.808768 20856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6: Bootstrap starting.
I20260924 15:22:40.809506 20856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:40.809841 20856 log.cc:824] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:40.810595 20856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6: No bootstrap required, opened a new log
I20260924 15:22:40.811194 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:40.809071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:50079 (local address 127.0.0.1:40129)
0924 15:22:40.809268 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:40.809276 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:40.809308 (+ 32us) server_negotiation.cc:407] Connection header received
0924 15:22:40.809949 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:40.809957 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:40.810000 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:40.810144 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:40.810778 (+ 634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:40.810782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:40.810782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:40.810786 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:40.810846 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:40.810856 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:40.810947 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:40.810992 (+ 45us) server_negotiation.cc:299] Negotiation successful
0924 15:22:40.811034 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
I20260924 15:22:40.812026 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:22:40.812819 20856 raft_consensus.cc:348] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943ab0911221424e8e8bc50354a7f5d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40129 } }
I20260924 15:22:40.813037 20856 raft_consensus.cc:374] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:40.813117 20856 raft_consensus.cc:729] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 943ab0911221424e8e8bc50354a7f5d6, State: Initialized, Role: FOLLOWER
I20260924 15:22:40.813269 20856 consensus_queue.cc:260] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943ab0911221424e8e8bc50354a7f5d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40129 } }
I20260924 15:22:40.813377 20856 raft_consensus.cc:388] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:40.813426 20856 raft_consensus.cc:482] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:40.813510 20856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:40.814268 20856 raft_consensus.cc:504] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943ab0911221424e8e8bc50354a7f5d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40129 } }
I20260924 15:22:40.814395 20856 leader_election.cc:302] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 943ab0911221424e8e8bc50354a7f5d6; no voters:
I20260924 15:22:40.814503 20859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:40.813376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:60771 (local address 127.0.0.1:40129)
0924 15:22:40.813416 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:40.813418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:40.813590 (+ 172us) server_negotiation.cc:407] Connection header received
0924 15:22:40.813658 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:40.813659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:40.813662 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:40.813708 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:40.814193 (+ 485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:40.814195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:40.814195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:40.814199 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:40.814211 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:40.814215 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:40.814271 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:40.814353 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 15:22:40.814395 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 15:22:40.814570 20856 leader_election.cc:288] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:40.814589 20866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:40.814795 20856 sys_catalog.cc:564] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:40.814857 20866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6 [term 1 LEADER]: Becoming Leader. State: Replica: 943ab0911221424e8e8bc50354a7f5d6, State: Running, Role: LEADER
I20260924 15:22:40.814930 20866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943ab0911221424e8e8bc50354a7f5d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40129 } }
I20260924 15:22:40.815287 20866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "943ab0911221424e8e8bc50354a7f5d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943ab0911221424e8e8bc50354a7f5d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40129 } } }
I20260924 15:22:40.815377 20866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:40.815522 20866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 943ab0911221424e8e8bc50354a7f5d6. Latest consensus state: current_term: 1 leader_uuid: "943ab0911221424e8e8bc50354a7f5d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "943ab0911221424e8e8bc50354a7f5d6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40129 } } }
I20260924 15:22:40.815608 20874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:40.815651 20866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:40.816236 20874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:40.817794 20874 catalog_manager.cc:1347] Generated new cluster ID: 4a436265da7a484d94ceb271a03b1d8d
I20260924 15:22:40.817843 20874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:40.826910 20874 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:40.827066 20874 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:40.835712 20874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 943ab0911221424e8e8bc50354a7f5d6: Generated new TSK 0
I20260924 15:22:40.835891 20874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:40.838670 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 20786
2026-09-24T15:22:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:40 dist-test-slave-zwbb krb5kdc[20887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:40 dist-test-slave-zwbb krb5kdc[20887](info): set up 2 sockets
Sep 24 15:22:40 dist-test-slave-zwbb krb5kdc[20887](info): commencing operation
krb5kdc: starting...
W20260924 15:22:41.863265 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:41 dist-test-slave-zwbb krb5kdc[20887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:22:41.976912 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35527
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35527
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:42.055300 20900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:42.055517 20900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:42.055557 20900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:42.056725 20900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:42.056782 20900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:42.056811 20900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:42.056842 20900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:42.056864 20900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:42.058389 20900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35527
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:42.058651 20900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:42.058888 20900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:42.063377 20906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:42.063386 20905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:42.063499 20900 server_base.cc:1034] running on GCE node
W20260924 15:22:42.063387 20908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:42.063831 20900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:42.064101 20900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:42.065266 20900 hybrid_clock.cc:648] HybridClock initialized: now 1790263362065267 us; error 45 us; skew 500 ppm
Sep 24 15:22:42 dist-test-slave-zwbb krb5kdc[20887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:42.067528 20900 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:22:42.067960 20900 webserver.cc:460] Webserver started at http://127.0.0.1:45305/ using document root <none> and password file <none>
I20260924 15:22:42.068192 20900 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:42.068252 20900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:42.068377 20900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:42.069322 20900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "b3a9b915889a43c0b526f052a1562097"
format_stamp: "Formatted at 2026-09-24 15:22:42 on dist-test-slave-zwbb"
server_key: "abd75d147a69d2d8f8b08367df771097"
server_key_iv: "a571d57888bed363e6bb4bc5d0c79937"
server_key_version: "encryptionkey@0"
I20260924 15:22:42.069725 20900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "b3a9b915889a43c0b526f052a1562097"
format_stamp: "Formatted at 2026-09-24 15:22:42 on dist-test-slave-zwbb"
server_key: "abd75d147a69d2d8f8b08367df771097"
server_key_iv: "a571d57888bed363e6bb4bc5d0c79937"
server_key_version: "encryptionkey@0"
I20260924 15:22:42.070995 20900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 15:22:42.071692 20916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:42.071877 20900 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:22:42.071972 20900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "b3a9b915889a43c0b526f052a1562097"
format_stamp: "Formatted at 2026-09-24 15:22:42 on dist-test-slave-zwbb"
server_key: "abd75d147a69d2d8f8b08367df771097"
server_key_iv: "a571d57888bed363e6bb4bc5d0c79937"
server_key_version: "encryptionkey@0"
I20260924 15:22:42.072065 20900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:42.079506 20900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:42.081218 20900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:42.081367 20900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:42.085309 20900 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35527
I20260924 15:22:42.085340 20969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35527 every 8 connection(s)
I20260924 15:22:42.085706 20900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:42.086558 20970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:42.088701 20970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097: Bootstrap starting.
I20260924 15:22:42.089273 20970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:42.089587 20970 log.cc:824] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:42.090257 20970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097: No bootstrap required, opened a new log
I20260924 15:22:42.091038 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 20900
I20260924 15:22:42.091126 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:42.091320 31793 external_mini_cluster.cc:1428] Setting key 81fd773e5043f8f2d29aa94df55d3abd
I20260924 15:22:42.092299 20970 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a9b915889a43c0b526f052a1562097" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } }
I20260924 15:22:42.092460 20970 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:42.092499 20970 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3a9b915889a43c0b526f052a1562097, State: Initialized, Role: FOLLOWER
I20260924 15:22:42.092641 20970 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a9b915889a43c0b526f052a1562097" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } }
I20260924 15:22:42.092731 20970 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:42.092752 20970 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:42.092798 20970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:42.093823 20970 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a9b915889a43c0b526f052a1562097" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } }
I20260924 15:22:42.093945 20970 leader_election.cc:302] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3a9b915889a43c0b526f052a1562097; no voters:
I20260924 15:22:42.094085 20970 leader_election.cc:288] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:42.094146 20975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:42.094307 20975 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097 [term 1 LEADER]: Becoming Leader. State: Replica: b3a9b915889a43c0b526f052a1562097, State: Running, Role: LEADER
I20260924 15:22:42.094311 20970 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:42.094427 20975 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a9b915889a43c0b526f052a1562097" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } }
I20260924 15:22:42.094810 20975 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3a9b915889a43c0b526f052a1562097. Latest consensus state: current_term: 1 leader_uuid: "b3a9b915889a43c0b526f052a1562097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a9b915889a43c0b526f052a1562097" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } } }
I20260924 15:22:42.094882 20975 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:42.095975 20976 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3a9b915889a43c0b526f052a1562097" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3a9b915889a43c0b526f052a1562097" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35527 } } }
I20260924 15:22:42.096053 20976 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097 [sys.catalog]: This master's current role is: LEADER
W20260924 15:22:42.096194 20991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:22:42.096254 20991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:22:42.096304 20982 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:42.096455 20982 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:42.098179 20982 catalog_manager.cc:1347] Generated new cluster ID: bd545bf57861446b8d3979c6043c83c6
I20260924 15:22:42.098229 20982 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:22:42 dist-test-slave-zwbb krb5kdc[20887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 15:22:42.102977 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:42.091802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:58339 (local address 127.0.0.1:35527)
0924 15:22:42.091994 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:42.092004 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:42.092042 (+ 38us) server_negotiation.cc:407] Connection header received
0924 15:22:42.092262 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:42.092270 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:42.092330 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:42.092476 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:42.093199 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:42.094621 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:42.095696 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:42.096324 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:42.096437 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:42.099333 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:42.099337 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:42.099337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:42.099348 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:42.101222 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:42.101737 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:42.101740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:42.101740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:42.101863 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:42.102113 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:42.102114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:42.102114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:42.102386 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:42.102553 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:42.102698 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 15:22:42.102792 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260924 15:22:42.104066 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:22:42.112936 20974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:42.105568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:49025 (local address 127.0.0.1:35527)
0924 15:22:42.105598 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:42.105599 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:42.105841 (+ 242us) server_negotiation.cc:407] Connection header received
0924 15:22:42.105896 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:42.105897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:42.105906 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:42.105957 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:42.106573 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:42.107292 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:42.108314 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:42.108619 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:42.108677 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:42.110064 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:42.110066 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:42.110066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:42.110072 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:42.111563 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:42.111973 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:42.111974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:42.111974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:42.112026 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:42.112242 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:42.112243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:42.112243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:42.112351 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:42.112449 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:42.112750 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 15:22:42.112831 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 15:22:42.117913 20982 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:42.118607 20982 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:42.128208 20982 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3a9b915889a43c0b526f052a1562097: Generated new TSK 0
I20260924 15:22:42.128386 20982 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:42.136462 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 20900
W20260924 15:22:42.141330 20885 connection.cc:535] client connection to 127.0.0.1:35527 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35527): Connection reset by peer (error 104)
2026-09-24T15:22:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:42 dist-test-slave-zwbb krb5kdc[21001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:42 dist-test-slave-zwbb krb5kdc[21001](info): set up 2 sockets
Sep 24 15:22:42 dist-test-slave-zwbb krb5kdc[21001](info): commencing operation
krb5kdc: starting...
W20260924 15:22:43.150933 31793 mini_kdc.cc:121] Time spent starting KDC: real 0.996s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:43 dist-test-slave-zwbb krb5kdc[21001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:22:43.264168 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35553
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35553
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:43.343500 21014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:43.343711 21014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:43.343753 21014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:43.344930 21014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:43.344998 21014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:43.345026 21014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:43.345057 21014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:43.345082 21014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:43.346581 21014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35553
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:43.346860 21014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:43.347075 21014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:43.351745 21019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:43.351788 21022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:43.351747 21020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:43.352047 21014 server_base.cc:1034] running on GCE node
I20260924 15:22:43.352305 21014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:43.352605 21014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:43.353775 21014 hybrid_clock.cc:648] HybridClock initialized: now 1790263363353764 us; error 39 us; skew 500 ppm
Sep 24 15:22:43 dist-test-slave-zwbb krb5kdc[21001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:43.356204 21014 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:22:43.356565 21014 webserver.cc:460] Webserver started at http://127.0.0.1:45059/ using document root <none> and password file <none>
I20260924 15:22:43.356755 21014 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:43.356805 21014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:43.356889 21014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:43.357914 21014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "389418734ad7461e9aa24bb7dd204d68"
format_stamp: "Formatted at 2026-09-24 15:22:43 on dist-test-slave-zwbb"
server_key: "a0030ffbe440cbd5092f79177b9c5dab"
server_key_iv: "9947af4c97feea9617fd82f6297f281f"
server_key_version: "encryptionkey@0"
I20260924 15:22:43.358225 21014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "389418734ad7461e9aa24bb7dd204d68"
format_stamp: "Formatted at 2026-09-24 15:22:43 on dist-test-slave-zwbb"
server_key: "a0030ffbe440cbd5092f79177b9c5dab"
server_key_iv: "9947af4c97feea9617fd82f6297f281f"
server_key_version: "encryptionkey@0"
I20260924 15:22:43.359508 21014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:22:43.360162 21029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:43.360349 21014 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:43.360426 21014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "389418734ad7461e9aa24bb7dd204d68"
format_stamp: "Formatted at 2026-09-24 15:22:43 on dist-test-slave-zwbb"
server_key: "a0030ffbe440cbd5092f79177b9c5dab"
server_key_iv: "9947af4c97feea9617fd82f6297f281f"
server_key_version: "encryptionkey@0"
I20260924 15:22:43.360491 21014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:43.383209 21014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:43.385475 21014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:43.385584 21014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:43.389180 21014 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35553
I20260924 15:22:43.389256 21082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35553 every 8 connection(s)
I20260924 15:22:43.389627 21014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:43.390480 21083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:43.392541 21083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68: Bootstrap starting.
I20260924 15:22:43.393093 21083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:43.393380 21083 log.cc:824] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:43.394069 21083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68: No bootstrap required, opened a new log
I20260924 15:22:43.395629 21083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389418734ad7461e9aa24bb7dd204d68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35553 } }
I20260924 15:22:43.395787 21083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:43.395850 21083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 389418734ad7461e9aa24bb7dd204d68, State: Initialized, Role: FOLLOWER
I20260924 15:22:43.396039 21083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389418734ad7461e9aa24bb7dd204d68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35553 } }
I20260924 15:22:43.396140 21083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:43.396185 21083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:43.396236 21083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:43.396723 21083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389418734ad7461e9aa24bb7dd204d68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35553 } }
I20260924 15:22:43.396838 21083 leader_election.cc:302] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 389418734ad7461e9aa24bb7dd204d68; no voters:
I20260924 15:22:43.397029 21083 leader_election.cc:288] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:43.397043 21086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:43.397205 21086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68 [term 1 LEADER]: Becoming Leader. State: Replica: 389418734ad7461e9aa24bb7dd204d68, State: Running, Role: LEADER
I20260924 15:22:43.397305 21086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389418734ad7461e9aa24bb7dd204d68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35553 } }
I20260924 15:22:43.397336 21083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:43.397732 21088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "389418734ad7461e9aa24bb7dd204d68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389418734ad7461e9aa24bb7dd204d68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35553 } } }
I20260924 15:22:43.397866 21087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 389418734ad7461e9aa24bb7dd204d68. Latest consensus state: current_term: 1 leader_uuid: "389418734ad7461e9aa24bb7dd204d68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "389418734ad7461e9aa24bb7dd204d68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35553 } } }
I20260924 15:22:43.397876 21088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:43.398010 21087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:43.398363 21093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:43.398880 21093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:43.399044 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 21014
I20260924 15:22:43.399163 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:43.399354 31793 external_mini_cluster.cc:1428] Setting key 8a2925d1ce6ae1ff2305533d51b67781
I20260924 15:22:43.400739 21093 catalog_manager.cc:1347] Generated new cluster ID: f8d1c67eb8b4483e996f8260c5762cac
I20260924 15:22:43.400816 21093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:43.405315 21106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:43.400021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:46355 (local address 127.0.0.1:35553)
0924 15:22:43.400171 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:43.400180 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:43.400213 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:22:43.400323 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:43.400331 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:43.400385 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:43.400523 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:43.400738 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:43.402128 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:43.403176 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:43.403803 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:43.403915 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:43.404602 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:43.404624 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:43.404624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:43.404830 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:43.404883 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:43.404897 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:43.404997 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:43.405076 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 15:22:43.405105 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":70,"threads_started":1}
I20260924 15:22:43.406607 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:22:43.411612 21106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:43.408147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:47297 (local address 127.0.0.1:35553)
0924 15:22:43.408187 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:43.408188 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:43.408332 (+ 144us) server_negotiation.cc:407] Connection header received
0924 15:22:43.408387 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:43.408388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:43.408398 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:43.408451 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:43.408568 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:43.409282 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:43.410334 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:43.410651 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:43.410710 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:43.411372 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:43.411373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:43.411374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:43.411378 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:43.411390 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:43.411394 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:43.411446 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:43.411529 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 15:22:43.411541 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:22:43.431769 21093 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:43.432488 21093 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:43.444348 21093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 389418734ad7461e9aa24bb7dd204d68: Generated new TSK 0
I20260924 15:22:43.444536 21093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:43.451952 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 21014
W20260924 15:22:43.456478 20999 connection.cc:535] client connection to 127.0.0.1:35553 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35553): Connection reset by peer (error 104)
2026-09-24T15:22:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:43 dist-test-slave-zwbb krb5kdc[21114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:43 dist-test-slave-zwbb krb5kdc[21114](info): set up 2 sockets
Sep 24 15:22:43 dist-test-slave-zwbb krb5kdc[21114](info): commencing operation
krb5kdc: starting...
W20260924 15:22:44.471408 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:44 dist-test-slave-zwbb krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263364, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 15:22:44.586321 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:44.665747 21127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:44.665941 21127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:44.665964 21127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:44.667042 21127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:44.667080 21127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:44.667094 21127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:44.667107 21127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:44.667119 21127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:44.668500 21127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39977
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:44.668679 21127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:44.668833 21127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:44.673580 21132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:44.673694 21127 server_base.cc:1034] running on GCE node
W20260924 15:22:44.673599 21134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:44.673591 21136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:44.674063 21127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:44.674372 21127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:44.675531 21127 hybrid_clock.cc:648] HybridClock initialized: now 1790263364675519 us; error 27 us; skew 500 ppm
Sep 24 15:22:44 dist-test-slave-zwbb krb5kdc[21114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:44.678341 21127 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 15:22:44.678958 21127 webserver.cc:460] Webserver started at http://127.0.0.1:37981/ using document root <none> and password file <none>
I20260924 15:22:44.679255 21127 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:44.679338 21127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:44.679476 21127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:44.680734 21127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "91491d7e79184af7b87f115b8f3062f5"
format_stamp: "Formatted at 2026-09-24 15:22:44 on dist-test-slave-zwbb"
server_key: "3860fce8b3cccf7d08c3c77d0a7a6d80"
server_key_iv: "dc5d4cdf968f48def518c219dba196a6"
server_key_version: "encryptionkey@0"
I20260924 15:22:44.681097 21127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "91491d7e79184af7b87f115b8f3062f5"
format_stamp: "Formatted at 2026-09-24 15:22:44 on dist-test-slave-zwbb"
server_key: "3860fce8b3cccf7d08c3c77d0a7a6d80"
server_key_iv: "dc5d4cdf968f48def518c219dba196a6"
server_key_version: "encryptionkey@0"
I20260924 15:22:44.682593 21127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:22:44.683298 21143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:44.683466 21127 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:22:44.683583 21127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "91491d7e79184af7b87f115b8f3062f5"
format_stamp: "Formatted at 2026-09-24 15:22:44 on dist-test-slave-zwbb"
server_key: "3860fce8b3cccf7d08c3c77d0a7a6d80"
server_key_iv: "dc5d4cdf968f48def518c219dba196a6"
server_key_version: "encryptionkey@0"
I20260924 15:22:44.683656 21127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:44.687036 21127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:44.688808 21127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:44.688963 21127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:44.693013 21127 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39977
I20260924 15:22:44.693064 21197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39977 every 8 connection(s)
I20260924 15:22:44.693403 21127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:44.694341 21198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:44.696974 21198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5: Bootstrap starting.
I20260924 15:22:44.697577 21198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:44.697815 21198 log.cc:824] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:44.698444 21198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5: No bootstrap required, opened a new log
I20260924 15:22:44.699926 21198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91491d7e79184af7b87f115b8f3062f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39977 } }
I20260924 15:22:44.700060 21198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:44.700081 21198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91491d7e79184af7b87f115b8f3062f5, State: Initialized, Role: FOLLOWER
I20260924 15:22:44.700176 21198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91491d7e79184af7b87f115b8f3062f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39977 } }
I20260924 15:22:44.700222 21198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:44.700241 21198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:44.700265 21198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:44.700680 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 21127
I20260924 15:22:44.700802 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:44.700847 21198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91491d7e79184af7b87f115b8f3062f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39977 } }
I20260924 15:22:44.700963 21198 leader_election.cc:302] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91491d7e79184af7b87f115b8f3062f5; no voters:
I20260924 15:22:44.700994 31793 external_mini_cluster.cc:1428] Setting key 124ad6c299e6e55722e9ed57205047aa
I20260924 15:22:44.701121 21198 leader_election.cc:288] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:44.701161 21201 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:44.701325 21201 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5 [term 1 LEADER]: Becoming Leader. State: Replica: 91491d7e79184af7b87f115b8f3062f5, State: Running, Role: LEADER
I20260924 15:22:44.701354 21198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:44.701500 21201 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91491d7e79184af7b87f115b8f3062f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39977 } }
I20260924 15:22:44.701887 21202 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91491d7e79184af7b87f115b8f3062f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91491d7e79184af7b87f115b8f3062f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39977 } } }
I20260924 15:22:44.701958 21202 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:44.701921 21203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91491d7e79184af7b87f115b8f3062f5. Latest consensus state: current_term: 1 leader_uuid: "91491d7e79184af7b87f115b8f3062f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91491d7e79184af7b87f115b8f3062f5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39977 } } }
I20260924 15:22:44.701999 21203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:44.702275 21207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:44.702847 21207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:44.703506 21205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:44.701603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:60575 (local address 127.0.0.1:39977)
0924 15:22:44.701770 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:44.701779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:44.701814 (+ 35us) server_negotiation.cc:407] Connection header received
0924 15:22:44.701945 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:44.701952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:44.701983 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:44.702378 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:44.702848 (+ 470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:44.702851 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:44.702851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:44.702856 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:44.702897 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:44.702912 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:44.703271 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:44.703300 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 15:22:44.703335 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":67,"threads_started":1}
I20260924 15:22:44.704432 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:22:44.704672 21207 catalog_manager.cc:1347] Generated new cluster ID: c654ef8099da42d3a70ef178e232c7a4
I20260924 15:22:44.704725 21207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:44.711438 21205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:44.709286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.1.8:48295 (local address 127.0.0.1:39977)
0924 15:22:44.709919 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:44.709920 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:44.709966 (+ 46us) server_negotiation.cc:407] Connection header received
0924 15:22:44.709987 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:44.709988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:44.709993 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:44.710072 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:44.711175 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:44.711176 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:44.711177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:44.711181 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:44.711194 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:44.711199 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:44.711229 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:44.711343 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 15:22:44.711358 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611}
I20260924 15:22:44.724064 21207 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:44.724231 21207 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:44.727475 21207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91491d7e79184af7b87f115b8f3062f5: Generated new TSK 0
I20260924 15:22:44.727659 21207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:44.737710 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 21127
2026-09-24T15:22:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1286 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:44 dist-test-slave-zwbb krb5kdc[21229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:44 dist-test-slave-zwbb krb5kdc[21229](info): set up 2 sockets
Sep 24 15:22:44 dist-test-slave-zwbb krb5kdc[21229](info): commencing operation
krb5kdc: starting...
W20260924 15:22:45.775056 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:45 dist-test-slave-zwbb krb5kdc[21229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:22:45.889755 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:41365
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:41001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:41365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:45.967849 21242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:45.968039 21242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:45.968061 21242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:45.969141 21242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:45.969177 21242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:45.969190 21242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:45.969202 21242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:45.969214 21242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:45.970597 21242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:41001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:41365
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:41365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:45.970772 21242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:45.970924 21242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:45.975505 21249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:45.975625 21251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:45.975625 21248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:45.975672 21242 server_base.cc:1034] running on GCE node
I20260924 15:22:45.975984 21242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:45.976230 21242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:45.977372 21242 hybrid_clock.cc:648] HybridClock initialized: now 1790263365977344 us; error 43 us; skew 500 ppm
Sep 24 15:22:45 dist-test-slave-zwbb krb5kdc[21229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263365, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:45.983700 21242 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:22:45.984089 21242 webserver.cc:460] Webserver started at http://127.31.12.126:34537/ using document root <none> and password file <none>
I20260924 15:22:45.984272 21242 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:45.984316 21242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:45.984403 21242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:45.985585 21242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "d6a2792ced6c4b05b3913115bddb0240"
format_stamp: "Formatted at 2026-09-24 15:22:45 on dist-test-slave-zwbb"
server_key: "b18b1c11407279f7d7a589ecc4b2ba60"
server_key_iv: "ee913d12bf41b1e498e6eeacd36eb0fb"
server_key_version: "encryptionkey@0"
I20260924 15:22:45.985895 21242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "d6a2792ced6c4b05b3913115bddb0240"
format_stamp: "Formatted at 2026-09-24 15:22:45 on dist-test-slave-zwbb"
server_key: "b18b1c11407279f7d7a589ecc4b2ba60"
server_key_iv: "ee913d12bf41b1e498e6eeacd36eb0fb"
server_key_version: "encryptionkey@0"
I20260924 15:22:45.987169 21242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:22:45.987805 21258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:45.987974 21242 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:45.988075 21242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "d6a2792ced6c4b05b3913115bddb0240"
format_stamp: "Formatted at 2026-09-24 15:22:45 on dist-test-slave-zwbb"
server_key: "b18b1c11407279f7d7a589ecc4b2ba60"
server_key_iv: "ee913d12bf41b1e498e6eeacd36eb0fb"
server_key_version: "encryptionkey@0"
I20260924 15:22:45.988158 21242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:45.998530 21242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:46.000202 21242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:46.000357 21242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:46.004528 21242 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:41365
I20260924 15:22:46.004575 21310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:41365 every 8 connection(s)
I20260924 15:22:46.004944 21242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:46.005893 21311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:46.008020 21311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240: Bootstrap starting.
I20260924 15:22:46.008517 21311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:46.008762 21311 log.cc:824] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:46.009384 21311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240: No bootstrap required, opened a new log
I20260924 15:22:46.010918 21311 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a2792ced6c4b05b3913115bddb0240" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 41365 } }
I20260924 15:22:46.011057 21311 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:46.011080 21311 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a2792ced6c4b05b3913115bddb0240, State: Initialized, Role: FOLLOWER
I20260924 15:22:46.011174 21311 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a2792ced6c4b05b3913115bddb0240" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 41365 } }
I20260924 15:22:46.011224 21311 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:46.011245 21311 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:46.011270 21311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:46.012040 21311 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a2792ced6c4b05b3913115bddb0240" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 41365 } }
I20260924 15:22:46.012123 21311 leader_election.cc:302] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6a2792ced6c4b05b3913115bddb0240; no voters:
I20260924 15:22:46.012259 21311 leader_election.cc:288] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:46.012322 21314 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:46.012485 21314 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240 [term 1 LEADER]: Becoming Leader. State: Replica: d6a2792ced6c4b05b3913115bddb0240, State: Running, Role: LEADER
I20260924 15:22:46.012521 21311 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:46.012607 21314 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a2792ced6c4b05b3913115bddb0240" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 41365 } }
I20260924 15:22:46.012982 21315 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6a2792ced6c4b05b3913115bddb0240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a2792ced6c4b05b3913115bddb0240" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 41365 } } }
I20260924 15:22:46.012993 21316 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6a2792ced6c4b05b3913115bddb0240. Latest consensus state: current_term: 1 leader_uuid: "d6a2792ced6c4b05b3913115bddb0240" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6a2792ced6c4b05b3913115bddb0240" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 41365 } } }
I20260924 15:22:46.013113 21315 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:46.013132 21316 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:46.013311 21323 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:46.013888 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 21242
I20260924 15:22:46.013937 21323 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:46.014010 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:46.014214 31793 external_mini_cluster.cc:1428] Setting key 9ba1363b6a5853ddfd8fa3c6ee98904a
I20260924 15:22:46.015640 21323 catalog_manager.cc:1347] Generated new cluster ID: 7e67618386904ced94ee392ca0f77003
I20260924 15:22:46.015681 21323 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:46.021396 21323 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:46.022100 21323 catalog_manager.cc:1504] Loading token signing keys...
Sep 24 15:22:46 dist-test-slave-zwbb krb5kdc[21229](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:22:46.027691 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:46.014644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45033 (local address 127.31.12.126:41365)
0924 15:22:46.017557 (+ 2913us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:46.017565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:46.017598 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:22:46.017703 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:46.017709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:46.017763 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:46.017893 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:46.018706 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:46.020119 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:46.021287 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:46.021825 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:46.021948 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:46.024318 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:46.024321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:46.024322 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:46.024333 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:46.026114 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:46.026525 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:46.026528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:46.026528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:46.026607 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:46.026844 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:46.026845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:46.026845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:46.027065 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:46.027210 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:46.027373 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 15:22:46.027474 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2865,"thread_start_us":64,"threads_started":1}
I20260924 15:22:46.028353 21323 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6a2792ced6c4b05b3913115bddb0240: Generated new TSK 0
I20260924 15:22:46.028636 21323 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:46.028861 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:22:46.036664 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:46.030357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40869 (local address 127.31.12.126:41365)
0924 15:22:46.030395 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:46.030396 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:46.030489 (+ 93us) server_negotiation.cc:407] Connection header received
0924 15:22:46.030547 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:46.030548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:46.030558 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:46.030620 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:46.031096 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:46.031914 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:46.032591 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:46.032915 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:46.032987 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:46.034093 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:46.034094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:46.034095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:46.034101 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:46.035588 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:46.035928 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:46.035929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:46.035929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:46.035989 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:46.036191 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:46.036192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:46.036192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:46.036315 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:46.036407 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:46.036514 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 15:22:46.036575 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:22:46.038534 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 21242
W20260924 15:22:46.042804 21227 connection.cc:535] client connection to 127.31.12.126:41365 recv error: Network error: failed to read from TLS socket (remote: 127.31.12.126:41365): Connection reset by peer (error 104)
2026-09-24T15:22:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:46 dist-test-slave-zwbb krb5kdc[21344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:46 dist-test-slave-zwbb krb5kdc[21344](info): set up 2 sockets
Sep 24 15:22:46 dist-test-slave-zwbb krb5kdc[21344](info): commencing operation
krb5kdc: starting...
W20260924 15:22:47.063388 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:47 dist-test-slave-zwbb krb5kdc[21344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:22:47.178393 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:44137
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:33515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:44137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:47.256809 21357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:47.257119 21357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:47.257156 21357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:47.258246 21357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:47.258285 21357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:47.258299 21357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:47.258311 21357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:47.258323 21357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:47.259639 21357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:33515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:44137
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:44137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:47.259811 21357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:47.259944 21357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:47.264551 21362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:47.264564 21363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:47.264564 21365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:47.264668 21357 server_base.cc:1034] running on GCE node
I20260924 15:22:47.265039 21357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:47.265266 21357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:47.266414 21357 hybrid_clock.cc:648] HybridClock initialized: now 1790263367266390 us; error 38 us; skew 500 ppm
Sep 24 15:22:47 dist-test-slave-zwbb krb5kdc[21344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263367, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:47.268864 21357 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:22:47.269282 21357 webserver.cc:460] Webserver started at http://127.31.12.126:35017/ using document root <none> and password file <none>
I20260924 15:22:47.269523 21357 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:47.269569 21357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:47.269660 21357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:47.270584 21357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "66a5541e363645e7b8ee5abd0ab580d9"
format_stamp: "Formatted at 2026-09-24 15:22:47 on dist-test-slave-zwbb"
server_key: "9cb775405151a05d13e340bcdd250a22"
server_key_iv: "8dcb329adeb0cd44e129feafd4d171f4"
server_key_version: "encryptionkey@0"
I20260924 15:22:47.270874 21357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "66a5541e363645e7b8ee5abd0ab580d9"
format_stamp: "Formatted at 2026-09-24 15:22:47 on dist-test-slave-zwbb"
server_key: "9cb775405151a05d13e340bcdd250a22"
server_key_iv: "8dcb329adeb0cd44e129feafd4d171f4"
server_key_version: "encryptionkey@0"
I20260924 15:22:47.272059 21357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:22:47.272691 21373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:47.272850 21357 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:22:47.272948 21357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "66a5541e363645e7b8ee5abd0ab580d9"
format_stamp: "Formatted at 2026-09-24 15:22:47 on dist-test-slave-zwbb"
server_key: "9cb775405151a05d13e340bcdd250a22"
server_key_iv: "8dcb329adeb0cd44e129feafd4d171f4"
server_key_version: "encryptionkey@0"
I20260924 15:22:47.273032 21357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:47.288874 21357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:47.290632 21357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:47.290783 21357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:47.294646 21357 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:44137
I20260924 15:22:47.294680 21425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:44137 every 8 connection(s)
I20260924 15:22:47.295024 21357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:47.295862 21426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:47.298003 21426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9: Bootstrap starting.
I20260924 15:22:47.298523 21426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:47.298851 21426 log.cc:824] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:47.299510 21426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9: No bootstrap required, opened a new log
I20260924 15:22:47.301023 21426 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a5541e363645e7b8ee5abd0ab580d9" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44137 } }
I20260924 15:22:47.301144 21426 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:47.301189 21426 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66a5541e363645e7b8ee5abd0ab580d9, State: Initialized, Role: FOLLOWER
I20260924 15:22:47.301335 21426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a5541e363645e7b8ee5abd0ab580d9" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44137 } }
I20260924 15:22:47.301393 21426 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:47.301429 21426 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:47.301498 21426 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:47.302238 21426 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a5541e363645e7b8ee5abd0ab580d9" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44137 } }
I20260924 15:22:47.302343 21426 leader_election.cc:302] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66a5541e363645e7b8ee5abd0ab580d9; no voters:
I20260924 15:22:47.302531 21426 leader_election.cc:288] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:47.302556 21429 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:47.302723 21429 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9 [term 1 LEADER]: Becoming Leader. State: Replica: 66a5541e363645e7b8ee5abd0ab580d9, State: Running, Role: LEADER
I20260924 15:22:47.302752 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 21357
I20260924 15:22:47.302856 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:47.302762 21426 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:47.302886 21429 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a5541e363645e7b8ee5abd0ab580d9" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44137 } }
I20260924 15:22:47.303061 31793 external_mini_cluster.cc:1428] Setting key b69d5f6a7b7b8a7739c96a96f70f2008
I20260924 15:22:47.303567 21431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66a5541e363645e7b8ee5abd0ab580d9. Latest consensus state: current_term: 1 leader_uuid: "66a5541e363645e7b8ee5abd0ab580d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a5541e363645e7b8ee5abd0ab580d9" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44137 } } }
I20260924 15:22:47.303663 21431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:47.303563 21430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66a5541e363645e7b8ee5abd0ab580d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66a5541e363645e7b8ee5abd0ab580d9" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44137 } } }
I20260924 15:22:47.303707 21430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:47.304272 21440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:47.304837 21440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:47.306361 21440 catalog_manager.cc:1347] Generated new cluster ID: 4df6b204224f489d8ef1c13628a9b882
I20260924 15:22:47.306413 21440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:47.309798 21439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:47.303659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58943 (local address 127.31.12.126:44137)
0924 15:22:47.304549 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:47.304559 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:47.304593 (+ 34us) server_negotiation.cc:407] Connection header received
0924 15:22:47.304713 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:47.304720 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:47.304775 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:47.304942 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:47.305194 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:47.306562 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:47.307652 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:47.308265 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:47.308377 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:47.309062 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:47.309066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:47.309066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:47.309264 (+ 198us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:47.309300 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:47.309315 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:47.309397 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:47.309544 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 15:22:47.309573 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":844,"thread_start_us":74,"threads_started":1}
I20260924 15:22:47.310694 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:22:47.312745 21440 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:47.313159 21440 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:47.315186 21439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:47.312197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42265 (local address 127.31.12.126:44137)
0924 15:22:47.312228 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:47.312230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:47.312238 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:22:47.312286 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:47.312286 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:47.312295 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:47.312350 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:47.312471 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:47.313123 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:47.314030 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:47.314302 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:47.314370 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:47.314925 (+ 555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:47.314926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:47.314927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:47.314930 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:47.314943 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:47.314947 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:47.315027 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:47.315106 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 15:22:47.315117 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:22:47.315676 21440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66a5541e363645e7b8ee5abd0ab580d9: Generated new TSK 0
I20260924 15:22:47.315814 21440 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:47.319490 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 21357
W20260924 15:22:47.324419 21342 connection.cc:535] client connection to 127.31.12.126:44137 recv error: Network error: failed to read from TLS socket (remote: 127.31.12.126:44137): Connection reset by peer (error 104)
2026-09-24T15:22:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1281 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:47 dist-test-slave-zwbb krb5kdc[21457](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:47 dist-test-slave-zwbb krb5kdc[21457](info): set up 2 sockets
Sep 24 15:22:47 dist-test-slave-zwbb krb5kdc[21457](info): commencing operation
krb5kdc: starting...
W20260924 15:22:48.355326 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:48 dist-test-slave-zwbb krb5kdc[21457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:22:48.472452 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:33101
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:46667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:33101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:48.553186 21471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:48.553397 21471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:48.553434 21471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:48.554565 21471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:48.554606 21471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:48.554621 21471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:48.554668 21471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:48.554695 21471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:48.556185 21471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:46667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:33101
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:33101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:48.556450 21471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:48.556620 21471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:48.561319 21478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:48.561364 21477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:48.561376 21471 server_base.cc:1034] running on GCE node
W20260924 15:22:48.561321 21480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:48.561803 21471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:48.562147 21471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:48.563308 21471 hybrid_clock.cc:648] HybridClock initialized: now 1790263368563292 us; error 46 us; skew 500 ppm
Sep 24 15:22:48 dist-test-slave-zwbb krb5kdc[21457](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263368, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:48.565682 21471 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:22:48.566099 21471 webserver.cc:460] Webserver started at http://127.31.12.126:43097/ using document root <none> and password file <none>
I20260924 15:22:48.566340 21471 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:48.566406 21471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:48.566566 21471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:48.567596 21471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "5a57f9f59cb940c990c19fcbc5f01897"
format_stamp: "Formatted at 2026-09-24 15:22:48 on dist-test-slave-zwbb"
server_key: "803ff94453af012ead3c37bac320cabe"
server_key_iv: "309a230835a6be3f2d3a79524d6a25dd"
server_key_version: "encryptionkey@0"
I20260924 15:22:48.567955 21471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "5a57f9f59cb940c990c19fcbc5f01897"
format_stamp: "Formatted at 2026-09-24 15:22:48 on dist-test-slave-zwbb"
server_key: "803ff94453af012ead3c37bac320cabe"
server_key_iv: "309a230835a6be3f2d3a79524d6a25dd"
server_key_version: "encryptionkey@0"
I20260924 15:22:48.569280 21471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:22:48.570065 21487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:48.570236 21471 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:48.570328 21471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "5a57f9f59cb940c990c19fcbc5f01897"
format_stamp: "Formatted at 2026-09-24 15:22:48 on dist-test-slave-zwbb"
server_key: "803ff94453af012ead3c37bac320cabe"
server_key_iv: "309a230835a6be3f2d3a79524d6a25dd"
server_key_version: "encryptionkey@0"
I20260924 15:22:48.570405 21471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:48.573510 21471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:48.575289 21471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:48.575449 21471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:48.579234 21471 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:33101
I20260924 15:22:48.579272 21539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:33101 every 8 connection(s)
I20260924 15:22:48.579577 21471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:48.580529 21540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:48.582613 21540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897: Bootstrap starting.
I20260924 15:22:48.583082 21540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:48.583294 21540 log.cc:824] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:48.583896 21540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897: No bootstrap required, opened a new log
I20260924 15:22:48.585402 21540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a57f9f59cb940c990c19fcbc5f01897" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33101 } }
I20260924 15:22:48.585608 21540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:48.585635 21540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a57f9f59cb940c990c19fcbc5f01897, State: Initialized, Role: FOLLOWER
I20260924 15:22:48.585736 21540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a57f9f59cb940c990c19fcbc5f01897" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33101 } }
I20260924 15:22:48.585786 21540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:48.585807 21540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:48.585834 21540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:48.586591 21540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a57f9f59cb940c990c19fcbc5f01897" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33101 } }
I20260924 15:22:48.586717 21540 leader_election.cc:302] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a57f9f59cb940c990c19fcbc5f01897; no voters:
I20260924 15:22:48.586826 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 21471
I20260924 15:22:48.586898 21540 leader_election.cc:288] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:48.586964 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:48.587064 21543 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:48.587139 21540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:48.587185 31793 external_mini_cluster.cc:1428] Setting key aa15d36e79852b0487161d90e90ae094
I20260924 15:22:48.587200 21543 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897 [term 1 LEADER]: Becoming Leader. State: Replica: 5a57f9f59cb940c990c19fcbc5f01897, State: Running, Role: LEADER
I20260924 15:22:48.587280 21543 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a57f9f59cb940c990c19fcbc5f01897" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33101 } }
I20260924 15:22:48.587636 21543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a57f9f59cb940c990c19fcbc5f01897. Latest consensus state: current_term: 1 leader_uuid: "5a57f9f59cb940c990c19fcbc5f01897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a57f9f59cb940c990c19fcbc5f01897" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33101 } } }
I20260924 15:22:48.587663 21544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a57f9f59cb940c990c19fcbc5f01897" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a57f9f59cb940c990c19fcbc5f01897" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 33101 } } }
I20260924 15:22:48.587809 21544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:48.588019 21543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:48.588035 21553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:48.588636 21553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:48.590343 21553 catalog_manager.cc:1347] Generated new cluster ID: ed7c7b0d6cd441808784afa029d85f2a
I20260924 15:22:48.590404 21553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:48.590778 21554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:48.587910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34785 (local address 127.31.12.126:33101)
0924 15:22:48.588596 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:48.588606 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:48.588639 (+ 33us) server_negotiation.cc:407] Connection header received
0924 15:22:48.588761 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:48.588769 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:48.588806 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:48.588952 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:48.590048 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:48.590051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:48.590052 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:48.590055 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:48.590086 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:48.590096 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:48.590306 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:48.590558 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 15:22:48.590599 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":62,"threads_started":1}
I20260924 15:22:48.591609 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:22:48.594547 21554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:48.592997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57997 (local address 127.31.12.126:33101)
0924 15:22:48.593027 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:48.593029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:48.593644 (+ 615us) server_negotiation.cc:407] Connection header received
0924 15:22:48.593708 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:48.593709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:48.593714 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:48.593763 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:48.594323 (+ 560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:48.594324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:48.594324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:48.594328 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:48.594342 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:48.594346 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:48.594398 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:48.594466 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 15:22:48.594478 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 15:22:48.598603 21553 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:48.598755 21553 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:48.609139 21553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a57f9f59cb940c990c19fcbc5f01897: Generated new TSK 0
I20260924 15:22:48.609308 21553 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:48.613049 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 21471
2026-09-24T15:22:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:48 dist-test-slave-zwbb krb5kdc[21571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:48 dist-test-slave-zwbb krb5kdc[21571](info): set up 2 sockets
Sep 24 15:22:48 dist-test-slave-zwbb krb5kdc[21571](info): commencing operation
krb5kdc: starting...
W20260924 15:22:49.639372 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:49 dist-test-slave-zwbb krb5kdc[21571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:22:49.750556 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:43785
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:35901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:43785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:49.829272 21584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:49.829607 21584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:49.829650 21584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:49.830827 21584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:49.830901 21584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:49.830927 21584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:49.830957 21584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:49.830979 21584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:49.832394 21584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:35901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:43785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:43785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:49.832670 21584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:49.832912 21584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:49.837657 21592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:49.837714 21590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:49.837666 21589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:49.837833 21584 server_base.cc:1034] running on GCE node
I20260924 15:22:49.838100 21584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:49.838384 21584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:49.839531 21584 hybrid_clock.cc:648] HybridClock initialized: now 1790263369839519 us; error 28 us; skew 500 ppm
Sep 24 15:22:49 dist-test-slave-zwbb krb5kdc[21571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263369, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:49.842072 21584 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:22:49.842510 21584 webserver.cc:460] Webserver started at http://127.31.12.126:33331/ using document root <none> and password file <none>
I20260924 15:22:49.842741 21584 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:49.842784 21584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:49.842967 21584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:49.843986 21584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "4f8fa247d2c84cf586900831dd8931ef"
format_stamp: "Formatted at 2026-09-24 15:22:49 on dist-test-slave-zwbb"
server_key: "6f7c143377aa3b5b8104f1fe5e3a6cd7"
server_key_iv: "abc8cb586cbdb757165a047ea7b67cdf"
server_key_version: "encryptionkey@0"
I20260924 15:22:49.844339 21584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "4f8fa247d2c84cf586900831dd8931ef"
format_stamp: "Formatted at 2026-09-24 15:22:49 on dist-test-slave-zwbb"
server_key: "6f7c143377aa3b5b8104f1fe5e3a6cd7"
server_key_iv: "abc8cb586cbdb757165a047ea7b67cdf"
server_key_version: "encryptionkey@0"
I20260924 15:22:49.845685 21584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:22:49.846369 21599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:49.846550 21584 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:49.846635 21584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "4f8fa247d2c84cf586900831dd8931ef"
format_stamp: "Formatted at 2026-09-24 15:22:49 on dist-test-slave-zwbb"
server_key: "6f7c143377aa3b5b8104f1fe5e3a6cd7"
server_key_iv: "abc8cb586cbdb757165a047ea7b67cdf"
server_key_version: "encryptionkey@0"
I20260924 15:22:49.846711 21584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:49.868026 21584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:49.870337 21584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:49.870477 21584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:49.874462 21584 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:43785
I20260924 15:22:49.874514 21653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:43785 every 8 connection(s)
I20260924 15:22:49.874895 21584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:49.875768 21654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:49.878026 21654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef: Bootstrap starting.
I20260924 15:22:49.878582 21654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:49.878860 21654 log.cc:824] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:49.879524 21654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef: No bootstrap required, opened a new log
I20260924 15:22:49.881058 21654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8fa247d2c84cf586900831dd8931ef" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43785 } }
I20260924 15:22:49.881198 21654 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:49.881218 21654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f8fa247d2c84cf586900831dd8931ef, State: Initialized, Role: FOLLOWER
I20260924 15:22:49.881371 21654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8fa247d2c84cf586900831dd8931ef" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43785 } }
I20260924 15:22:49.881426 21654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:49.881505 21654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:49.881553 21654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:49.882314 21654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8fa247d2c84cf586900831dd8931ef" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43785 } }
I20260924 15:22:49.882440 21654 leader_election.cc:302] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f8fa247d2c84cf586900831dd8931ef; no voters:
I20260924 15:22:49.882632 21654 leader_election.cc:288] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:49.882660 21657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:49.882838 21657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef [term 1 LEADER]: Becoming Leader. State: Replica: 4f8fa247d2c84cf586900831dd8931ef, State: Running, Role: LEADER
I20260924 15:22:49.882920 21654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:49.882959 21657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8fa247d2c84cf586900831dd8931ef" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43785 } }
I20260924 15:22:49.883363 21659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f8fa247d2c84cf586900831dd8931ef. Latest consensus state: current_term: 1 leader_uuid: "4f8fa247d2c84cf586900831dd8931ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8fa247d2c84cf586900831dd8931ef" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43785 } } }
I20260924 15:22:49.883361 21658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f8fa247d2c84cf586900831dd8931ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f8fa247d2c84cf586900831dd8931ef" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 43785 } } }
I20260924 15:22:49.883553 21659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:49.883563 21658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:49.884146 21666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:49.884667 21666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:49.885082 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 21584
I20260924 15:22:49.885205 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:49.885378 31793 external_mini_cluster.cc:1428] Setting key 45563e195d801171ab2edbd4741046fd
I20260924 15:22:49.889887 21666 catalog_manager.cc:1347] Generated new cluster ID: 883bc52c010d4933aab7e77f46a890a9
I20260924 15:22:49.889943 21666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 15:22:49 dist-test-slave-zwbb krb5kdc[21571](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:22:49.902920 21677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:49.885891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52077 (local address 127.31.12.126:43785)
0924 15:22:49.889639 (+ 3748us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:49.889646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:49.889672 (+ 26us) server_negotiation.cc:407] Connection header received
0924 15:22:49.889760 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:49.889765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:49.892663 (+ 2898us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:49.892802 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:49.893621 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:49.894930 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:49.896034 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:49.896615 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:49.896710 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:49.899363 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:49.899368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:49.899368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:49.899381 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:49.901298 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:49.901723 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:49.901727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:49.901727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:49.901834 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:49.902048 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:49.902048 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:49.902049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:49.902354 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:49.902512 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:49.902585 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 15:22:49.902714 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3690,"thread_start_us":69,"threads_started":1}
I20260924 15:22:49.903816 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:22:49.906280 21666 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:49.906944 21666 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:49.911413 21677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:49.905306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33517 (local address 127.31.12.126:43785)
0924 15:22:49.905340 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:49.905341 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:49.905455 (+ 114us) server_negotiation.cc:407] Connection header received
0924 15:22:49.905507 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:49.905507 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:49.905517 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:49.905553 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:49.906265 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:49.906780 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:49.907460 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:49.907670 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:49.907753 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:49.908885 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:49.908886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:49.908886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:49.908893 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:49.910263 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:49.910612 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:49.910613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:49.910614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:49.910684 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:49.910875 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:49.910876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:49.910876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:49.910980 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:49.911102 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:49.911190 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 15:22:49.911288 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:22:49.913861 21666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f8fa247d2c84cf586900831dd8931ef: Generated new TSK 0
I20260924 15:22:49.914040 21666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:49.918401 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 21584
2026-09-24T15:22:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:49 dist-test-slave-zwbb krb5kdc[21685](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:49 dist-test-slave-zwbb krb5kdc[21685](info): set up 2 sockets
Sep 24 15:22:49 dist-test-slave-zwbb krb5kdc[21685](info): commencing operation
krb5kdc: starting...
W20260924 15:22:51.003437 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:51 dist-test-slave-zwbb krb5kdc[21685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:22:51.118373 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:45569
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:45535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:45569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:51.198489 21698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:51.198695 21698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:51.198719 21698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:51.199890 21698 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:51.199932 21698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:51.199946 21698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:51.199958 21698 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:51.199971 21698 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:51.201473 21698 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:45535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:45569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:45569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:51.201721 21698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:51.201947 21698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:51.206810 21703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:51.206825 21704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:51.206853 21706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:51.206979 21698 server_base.cc:1034] running on GCE node
I20260924 15:22:51.207289 21698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:51.207597 21698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:51.208763 21698 hybrid_clock.cc:648] HybridClock initialized: now 1790263371208743 us; error 33 us; skew 500 ppm
Sep 24 15:22:51 dist-test-slave-zwbb krb5kdc[21685](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263371, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:51.216001 21698 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:22:51.216460 21698 webserver.cc:460] Webserver started at http://127.31.12.126:37885/ using document root <none> and password file <none>
I20260924 15:22:51.216651 21698 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:51.216691 21698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:51.216773 21698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:51.217823 21698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "b5ca7e91f5e048beb3b406fa9d4cb087"
format_stamp: "Formatted at 2026-09-24 15:22:51 on dist-test-slave-zwbb"
server_key: "87c3770b291743c379865721479533eb"
server_key_iv: "d075afcc03e8964f10392f269386cc9f"
server_key_version: "encryptionkey@0"
I20260924 15:22:51.218113 21698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "b5ca7e91f5e048beb3b406fa9d4cb087"
format_stamp: "Formatted at 2026-09-24 15:22:51 on dist-test-slave-zwbb"
server_key: "87c3770b291743c379865721479533eb"
server_key_iv: "d075afcc03e8964f10392f269386cc9f"
server_key_version: "encryptionkey@0"
I20260924 15:22:51.219306 21698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:22:51.219970 21713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:51.220160 21698 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:51.220242 21698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "b5ca7e91f5e048beb3b406fa9d4cb087"
format_stamp: "Formatted at 2026-09-24 15:22:51 on dist-test-slave-zwbb"
server_key: "87c3770b291743c379865721479533eb"
server_key_iv: "d075afcc03e8964f10392f269386cc9f"
server_key_version: "encryptionkey@0"
I20260924 15:22:51.220295 21698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:51.245128 21698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:51.247514 21698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:51.247629 21698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:51.251514 21698 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:45569
I20260924 15:22:51.251591 21767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:45569 every 8 connection(s)
I20260924 15:22:51.251940 21698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:51.252822 21768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:51.253202 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 21698
I20260924 15:22:51.253332 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:51.253573 31793 external_mini_cluster.cc:1428] Setting key ade95d21033d69e953ac7d0b6dbf19c1
I20260924 15:22:51.255781 21768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087: Bootstrap starting.
I20260924 15:22:51.256444 21768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:51.256788 21768 log.cc:824] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:51.257567 21768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087: No bootstrap required, opened a new log
I20260924 15:22:51.259327 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:51.254116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42061 (local address 127.31.12.126:45569)
0924 15:22:51.254300 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:51.254310 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:51.254341 (+ 31us) server_negotiation.cc:407] Connection header received
0924 15:22:51.254754 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:51.254761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:51.254805 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:51.254916 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:51.255189 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:51.256148 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:51.257288 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:51.257773 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:51.257853 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:51.258481 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:51.258485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:51.258485 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:51.258699 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:51.258728 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:51.258738 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:51.258786 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:51.258886 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 15:22:51.258942 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260924 15:22:51.259443 21768 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ca7e91f5e048beb3b406fa9d4cb087" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 45569 } }
I20260924 15:22:51.259630 21768 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:51.259676 21768 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5ca7e91f5e048beb3b406fa9d4cb087, State: Initialized, Role: FOLLOWER
I20260924 15:22:51.259795 21768 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ca7e91f5e048beb3b406fa9d4cb087" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 45569 } }
I20260924 15:22:51.259891 21768 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:51.259966 21768 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:51.260023 21768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:51.260284 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:22:51.260699 21768 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ca7e91f5e048beb3b406fa9d4cb087" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 45569 } }
I20260924 15:22:51.260820 21768 leader_election.cc:302] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5ca7e91f5e048beb3b406fa9d4cb087; no voters:
I20260924 15:22:51.260990 21768 leader_election.cc:288] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:51.261050 21773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:51.261185 21773 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087 [term 1 LEADER]: Becoming Leader. State: Replica: b5ca7e91f5e048beb3b406fa9d4cb087, State: Running, Role: LEADER
I20260924 15:22:51.261235 21768 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:51.261319 21773 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ca7e91f5e048beb3b406fa9d4cb087" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 45569 } }
I20260924 15:22:51.261775 21775 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5ca7e91f5e048beb3b406fa9d4cb087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ca7e91f5e048beb3b406fa9d4cb087" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 45569 } } }
I20260924 15:22:51.261942 21775 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:51.262149 21775 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5ca7e91f5e048beb3b406fa9d4cb087. Latest consensus state: current_term: 1 leader_uuid: "b5ca7e91f5e048beb3b406fa9d4cb087" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5ca7e91f5e048beb3b406fa9d4cb087" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 45569 } } }
I20260924 15:22:51.262208 21787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:51.262261 21775 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:51.262737 21787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:51.265105 21787 catalog_manager.cc:1347] Generated new cluster ID: 0751f2181fcf4a01be5502e5e3cb8629
I20260924 15:22:51.265151 21787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:51.265889 21771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:51.261794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44719 (local address 127.31.12.126:45569)
0924 15:22:51.261902 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:51.261904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:51.262839 (+ 935us) server_negotiation.cc:407] Connection header received
0924 15:22:51.262938 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:51.262939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:51.262949 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:51.262978 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:51.263150 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:51.263728 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:51.264690 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:51.264878 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:51.264967 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:51.265627 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:51.265628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:51.265629 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:51.265633 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:51.265647 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:51.265652 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:51.265697 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:51.265794 (+ 97us) server_negotiation.cc:299] Negotiation successful
0924 15:22:51.265814 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 15:22:51.273036 21787 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:51.273703 21787 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:51.279678 21787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5ca7e91f5e048beb3b406fa9d4cb087: Generated new TSK 0
I20260924 15:22:51.279850 21787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:51.282897 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 21698
2026-09-24T15:22:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:51 dist-test-slave-zwbb krb5kdc[21799](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:51 dist-test-slave-zwbb krb5kdc[21799](info): set up 2 sockets
Sep 24 15:22:51 dist-test-slave-zwbb krb5kdc[21799](info): commencing operation
krb5kdc: starting...
W20260924 15:22:52.307233 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:52 dist-test-slave-zwbb krb5kdc[21799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:22:52.422576 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:44447
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:35521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:44447
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:52.499732 21812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:52.499923 21812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:52.499946 21812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:52.501134 21812 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:52.501173 21812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:52.501187 21812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:52.501200 21812 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:52.501214 21812 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:52.502781 21812 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:35521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:44447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:44447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:52.503017 21812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:52.503269 21812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:52.507949 21820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:52.507970 21818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:52.507949 21817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:52.508208 21812 server_base.cc:1034] running on GCE node
I20260924 15:22:52.508471 21812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:52.508740 21812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:52.509884 21812 hybrid_clock.cc:648] HybridClock initialized: now 1790263372509873 us; error 48 us; skew 500 ppm
Sep 24 15:22:52 dist-test-slave-zwbb krb5kdc[21799](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263372, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:52.512244 21812 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:22:52.512670 21812 webserver.cc:460] Webserver started at http://127.31.12.126:39715/ using document root <none> and password file <none>
I20260924 15:22:52.512881 21812 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:52.512943 21812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:52.513077 21812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:52.514057 21812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "789f6c94b9bf4a0e9de33473133832e3"
format_stamp: "Formatted at 2026-09-24 15:22:52 on dist-test-slave-zwbb"
server_key: "3637bc51e063f895a554cbbcace2cdf2"
server_key_iv: "96a22cf302d1ae70e857d4cc722bd274"
server_key_version: "encryptionkey@0"
I20260924 15:22:52.514386 21812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "789f6c94b9bf4a0e9de33473133832e3"
format_stamp: "Formatted at 2026-09-24 15:22:52 on dist-test-slave-zwbb"
server_key: "3637bc51e063f895a554cbbcace2cdf2"
server_key_iv: "96a22cf302d1ae70e857d4cc722bd274"
server_key_version: "encryptionkey@0"
I20260924 15:22:52.515656 21812 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:22:52.516458 21827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:52.516675 21812 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 15:22:52.516778 21812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "789f6c94b9bf4a0e9de33473133832e3"
format_stamp: "Formatted at 2026-09-24 15:22:52 on dist-test-slave-zwbb"
server_key: "3637bc51e063f895a554cbbcace2cdf2"
server_key_iv: "96a22cf302d1ae70e857d4cc722bd274"
server_key_version: "encryptionkey@0"
I20260924 15:22:52.516850 21812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:52.519987 21812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:52.522001 21812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:52.522130 21812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:52.525945 21812 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:44447
I20260924 15:22:52.525977 21879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:44447 every 8 connection(s)
I20260924 15:22:52.526264 21812 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:52.526427 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 21812
I20260924 15:22:52.526557 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:52.526788 31793 external_mini_cluster.cc:1428] Setting key 1c1d967bca49d2bf8f7ee19686c8e7d8
I20260924 15:22:52.527521 21880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:52.529045 21883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:52.527286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59173 (local address 127.31.12.126:44447)
0924 15:22:52.527450 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:52.527458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:52.527489 (+ 31us) server_negotiation.cc:407] Connection header received
0924 15:22:52.527918 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:52.527924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:52.527947 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:52.528053 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:52.528605 (+ 552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:52.528608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:52.528608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:52.528611 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:52.528644 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:52.528653 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:52.528703 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:52.528796 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 15:22:52.528835 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260924 15:22:52.529986 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 15:22:52.530674 21880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3: Bootstrap starting.
I20260924 15:22:52.531394 21880 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:52.531728 21880 log.cc:824] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:52.532378 21880 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3: No bootstrap required, opened a new log
I20260924 15:22:52.532487 21883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:52.531585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34021 (local address 127.31.12.126:44447)
0924 15:22:52.531621 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:52.531623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:52.531724 (+ 101us) server_negotiation.cc:407] Connection header received
0924 15:22:52.531785 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:52.531786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:52.531790 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:52.531830 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:52.532283 (+ 453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:52.532284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:52.532284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 15:22:52.532288 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:52.532298 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 15:22:52.532301 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:52.532348 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:52.532421 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 15:22:52.532433 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:22:52.534500 21880 raft_consensus.cc:348] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789f6c94b9bf4a0e9de33473133832e3" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44447 } }
I20260924 15:22:52.534651 21880 raft_consensus.cc:374] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 15:22:52.534699 21880 raft_consensus.cc:729] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789f6c94b9bf4a0e9de33473133832e3, State: Initialized, Role: FOLLOWER
I20260924 15:22:52.534811 21880 consensus_queue.cc:260] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789f6c94b9bf4a0e9de33473133832e3" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44447 } }
I20260924 15:22:52.534924 21880 raft_consensus.cc:388] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 15:22:52.534974 21880 raft_consensus.cc:482] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 15:22:52.535022 21880 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:52.535686 21880 raft_consensus.cc:504] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789f6c94b9bf4a0e9de33473133832e3" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44447 } }
I20260924 15:22:52.535812 21880 leader_election.cc:302] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 789f6c94b9bf4a0e9de33473133832e3; no voters:
I20260924 15:22:52.535986 21880 leader_election.cc:288] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 15:22:52.536034 21890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:52.536211 21890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3 [term 1 LEADER]: Becoming Leader. State: Replica: 789f6c94b9bf4a0e9de33473133832e3, State: Running, Role: LEADER
I20260924 15:22:52.536242 21880 sys_catalog.cc:564] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:52.536361 21890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789f6c94b9bf4a0e9de33473133832e3" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44447 } }
I20260924 15:22:52.536756 21892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 789f6c94b9bf4a0e9de33473133832e3. Latest consensus state: current_term: 1 leader_uuid: "789f6c94b9bf4a0e9de33473133832e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789f6c94b9bf4a0e9de33473133832e3" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44447 } } }
I20260924 15:22:52.536754 21891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "789f6c94b9bf4a0e9de33473133832e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "789f6c94b9bf4a0e9de33473133832e3" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 44447 } } }
I20260924 15:22:52.536892 21892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:52.537009 21891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:52.537343 21898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:52.537876 21898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:52.539356 21898 catalog_manager.cc:1347] Generated new cluster ID: 8e081bcd0daf4c79a23e3d1a25000cb0
I20260924 15:22:52.539419 21898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:52.550647 21898 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:52.550798 21898 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:52.557241 21898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 789f6c94b9bf4a0e9de33473133832e3: Generated new TSK 0
I20260924 15:22:52.557417 21898 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:52.562052 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 21812
2026-09-24T15:22:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1278 ms)
[----------] 18 tests from AuthTokenIssuingTest (23479 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:52 dist-test-slave-zwbb krb5kdc[21912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:52 dist-test-slave-zwbb krb5kdc[21912](info): set up 2 sockets
Sep 24 15:22:52 dist-test-slave-zwbb krb5kdc[21912](info): commencing operation
krb5kdc: starting...
W20260924 15:22:53.578487 31793 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:53 dist-test-slave-zwbb krb5kdc[21912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:22:53.692128 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:35957
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:38383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:35957,127.31.12.125:37555,127.31.12.124:39671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:53.780879 21926 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 15:22:53.781057 21926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:53.781072 21926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:53.781091 21926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:53.782148 21926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:53.782183 21926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:53.782196 21926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:53.782241 21926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:53.782258 21926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:53.783675 21926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:38383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:35957,127.31.12.125:37555,127.31.12.124:39671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:35957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:53.783929 21926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:53.784149 21926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:53.788738 21931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:53.788846 21926 server_base.cc:1034] running on GCE node
W20260924 15:22:53.788767 21934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:53.788756 21932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:53.789149 21926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:53.789426 21926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:53.790634 21926 hybrid_clock.cc:648] HybridClock initialized: now 1790263373790633 us; error 38 us; skew 500 ppm
Sep 24 15:22:53 dist-test-slave-zwbb krb5kdc[21912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263373, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:53.792815 21926 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:22:53.793186 21926 webserver.cc:460] Webserver started at http://127.31.12.126:45877/ using document root <none> and password file <none>
I20260924 15:22:53.793386 21926 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:53.793447 21926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:53.793608 21926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:53.794528 21926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "16ccbeeac04a4006ab5326ea02841007"
format_stamp: "Formatted at 2026-09-24 15:22:53 on dist-test-slave-zwbb"
server_key: "3ca80c8215213c5bce93b286d61adc8e"
server_key_iv: "13e764341623178173454b378bb3505c"
server_key_version: "encryptionkey@0"
I20260924 15:22:53.794889 21926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "16ccbeeac04a4006ab5326ea02841007"
format_stamp: "Formatted at 2026-09-24 15:22:53 on dist-test-slave-zwbb"
server_key: "3ca80c8215213c5bce93b286d61adc8e"
server_key_iv: "13e764341623178173454b378bb3505c"
server_key_version: "encryptionkey@0"
I20260924 15:22:53.796167 21926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 15:22:53.796895 21941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:53.797052 21926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:53.797140 21926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "16ccbeeac04a4006ab5326ea02841007"
format_stamp: "Formatted at 2026-09-24 15:22:53 on dist-test-slave-zwbb"
server_key: "3ca80c8215213c5bce93b286d61adc8e"
server_key_iv: "13e764341623178173454b378bb3505c"
server_key_version: "encryptionkey@0"
I20260924 15:22:53.797214 21926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:53.805672 21926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:53.807367 21926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:53.807523 21926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:53.811285 21926 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:35957
I20260924 15:22:53.811313 21994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:35957 every 8 connection(s)
I20260924 15:22:53.811625 21926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:53.812479 21995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:53.813858 21995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:22:53.816572 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 21926
I20260924 15:22:53.816685 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:53.816883 31793 external_mini_cluster.cc:1428] Setting key 168226a83f0b1671e4b998acfc30f6a4
Sep 24 15:22:53 dist-test-slave-zwbb krb5kdc[21912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263373, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:22:53.823117 21998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:53.814585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40232 (local address 127.31.12.126:35957)
0924 15:22:53.814693 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:53.814697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:53.814728 (+ 31us) server_negotiation.cc:407] Connection header received
0924 15:22:53.815267 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:53.815269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:53.815285 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:53.815379 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:53.815880 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:53.816742 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:53.817655 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:53.817937 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:53.818047 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:53.820093 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:53.820100 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:53.820100 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:53.820106 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:53.821599 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:53.821962 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:53.821965 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:53.821965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:53.822040 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:53.822264 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:53.822265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:53.822265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:53.822467 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:53.822556 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:53.822761 (+ 205us) server_negotiation.cc:299] Negotiation successful
0924 15:22:53.822902 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":81,"thread_start_us":41,"threads_started":1}
I20260924 15:22:53.823119 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:53.814443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:35957 (local address 127.0.0.1:40232)
0924 15:22:53.814592 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:53.814640 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:53.815104 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:53.815385 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:53.815388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:53.815647 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:53.815780 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:53.815784 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:53.816831 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:53.816832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:53.817560 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:53.817563 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:53.818046 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:53.818047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:53.818110 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:53.818546 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:53.818554 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:53.820010 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:53.821685 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:53.821691 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:53.821693 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:53.821872 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:53.822118 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:53.822118 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:53.822119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:53.822205 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:53.822551 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:53.822556 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:53.822614 (+ 58us) client_negotiation.cc:769] Sending connection context
0924 15:22:53.822716 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 15:22:53.822945 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":114,"thread_start_us":68,"threads_started":1}
I20260924 15:22:53.823978 21995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.12.125@KRBTEST.COM; defaulting to no policy
I20260924 15:22:53.824292 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:53.824190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:37555 (local address 127.0.0.1:45462)
0924 15:22:53.824237 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:53.824274 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.125:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260924 15:22:53.824414 21945 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.12.125:37555: connect: Connection refused (error 111)
I20260924 15:22:53.824781 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:53.824691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:37555 (local address 127.0.0.1:45476)
0924 15:22:53.824739 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:53.824761 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.125:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 15:22:53.824935 21995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.125:37555: Network error: Client connection negotiation failed: client connection to 127.31.12.125:37555: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.125.keytab.
Entry for principal kudu/127.31.12.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.125.keytab.
I20260924 15:22:53.840993 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.125:37555
--webserver_interface=127.31.12.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.125.keytab
--principal=kudu/127.31.12.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:38383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:35957,127.31.12.125:37555,127.31.12.124:39671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:22:53.874182 21995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } attempt: 1
I20260924 15:22:53.874656 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:53.874522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:37555 (local address 127.0.0.1:45482)
0924 15:22:53.874588 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:53.874628 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.125:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 15:22:53.875043 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:53.874913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:37555 (local address 127.0.0.1:45488)
0924 15:22:53.874970 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:53.875013 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.125:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 15:22:53.875221 21995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.125:37555: Network error: Client connection negotiation failed: client connection to 127.31.12.125:37555: connect: Connection refused (error 111)
W20260924 15:22:53.919785 22002 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 15:22:53.919987 22002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:53.920004 22002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:53.920023 22002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:53.921154 22002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:53.921193 22002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:53.921207 22002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:53.921221 22002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:53.921233 22002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:53.922755 22002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:38383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:35957,127.31.12.125:37555,127.31.12.124:39671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.125.keytab
--principal=kudu/127.31.12.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.125:37555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.12.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:53.923003 22002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:53.923219 22002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:53.928074 22007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:53.928126 22010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:53.928229 22002 server_base.cc:1034] running on GCE node
W20260924 15:22:53.928301 22008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:53.928532 22002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:53.928848 22002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:53.930006 22002 hybrid_clock.cc:648] HybridClock initialized: now 1790263373929996 us; error 46 us; skew 500 ppm
Sep 24 15:22:53 dist-test-slave-zwbb krb5kdc[21912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263373, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:53.932345 22002 init.cc:377] Logged in from keytab as kudu/127.31.12.125@KRBTEST.COM (short username kudu)
I20260924 15:22:53.932796 22002 webserver.cc:460] Webserver started at http://127.31.12.125:39075/ using document root <none> and password file <none>
I20260924 15:22:53.933010 22002 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:53.933084 22002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:53.933229 22002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:53.934299 22002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-1/data/instance:
uuid: "6a8c5d13200847248f80cdbf527520ef"
format_stamp: "Formatted at 2026-09-24 15:22:53 on dist-test-slave-zwbb"
server_key: "e36b54e6f6edcf21ee14a6f7fb7f2473"
server_key_iv: "24aa58318d8fb3aa3e9813413f58d314"
server_key_version: "encryptionkey@0"
I20260924 15:22:53.934662 22002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-1/wal/instance:
uuid: "6a8c5d13200847248f80cdbf527520ef"
format_stamp: "Formatted at 2026-09-24 15:22:53 on dist-test-slave-zwbb"
server_key: "e36b54e6f6edcf21ee14a6f7fb7f2473"
server_key_iv: "24aa58318d8fb3aa3e9813413f58d314"
server_key_version: "encryptionkey@0"
I20260924 15:22:53.935977 22002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:22:53.936713 22017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:53.936892 22002 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:22:53.936991 22002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-1/wal
uuid: "6a8c5d13200847248f80cdbf527520ef"
format_stamp: "Formatted at 2026-09-24 15:22:53 on dist-test-slave-zwbb"
server_key: "e36b54e6f6edcf21ee14a6f7fb7f2473"
server_key_iv: "24aa58318d8fb3aa3e9813413f58d314"
server_key_version: "encryptionkey@0"
I20260924 15:22:53.937089 22002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:53.943387 21995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } attempt: 2
I20260924 15:22:53.943778 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:53.943667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:37555 (local address 127.0.0.1:45490)
0924 15:22:53.943719 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:53.943753 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.125:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 15:22:53.944170 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:53.944074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:37555 (local address 127.0.0.1:45504)
0924 15:22:53.944118 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:53.944150 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.125:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 15:22:53.944257 21995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.125:37555: Network error: Client connection negotiation failed: client connection to 127.31.12.125:37555: connect: Connection refused (error 111)
I20260924 15:22:53.962985 22002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:53.965874 22002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:53.966050 22002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:53.970067 22002 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.125:37555
I20260924 15:22:53.970096 22071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.125:37555 every 8 connection(s)
I20260924 15:22:53.970440 22002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-1/data/info.pb
I20260924 15:22:53.971356 22072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:53.972824 22072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:22:53.975585 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 22002
I20260924 15:22:53.975693 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-1/wal/instance
I20260924 15:22:53.975922 31793 external_mini_cluster.cc:1428] Setting key c9417eccdcc7e50bc43e8cddd1550e59
Sep 24 15:22:53 dist-test-slave-zwbb krb5kdc[21912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263373, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.125@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:22:53.982467 21998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:53.973569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.31.12.126:35957)
0924 15:22:53.973634 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:53.973637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:53.973869 (+ 232us) server_negotiation.cc:407] Connection header received
0924 15:22:53.974379 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:53.974381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:53.974394 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:53.974474 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:53.975038 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:53.975526 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:53.976975 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:53.977174 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:53.977229 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:53.979632 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:53.979634 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:53.979635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:53.979643 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:53.981065 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:53.981474 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:53.981476 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:53.981476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:53.981540 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:53.981799 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:53.981800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:53.981800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:53.981910 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:53.981996 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:53.982261 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 15:22:53.982357 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:22:53.982689 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:53.973537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:35957 (local address 127.0.0.1:40238)
0924 15:22:53.973740 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:53.973779 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:53.974263 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:53.974521 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:53.974524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:53.974765 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:53.974963 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:53.974967 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:53.975623 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:53.975624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:53.976891 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:53.976895 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:53.977278 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:53.977279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:53.977466 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:53.978051 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:53.978067 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:53.979530 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:53.981149 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:53.981156 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:53.981159 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:53.981383 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:53.981609 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:53.981610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:53.981611 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:53.981724 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:53.981978 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:53.981984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:53.982127 (+ 143us) client_negotiation.cc:769] Sending connection context
0924 15:22:53.982229 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 15:22:53.982411 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":113,"threads_started":1}
I20260924 15:22:53.983258 22072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.31.12.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.124@KRBTEST.COM" created.
Sep 24 15:22:53 dist-test-slave-zwbb krb5kdc[21912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263373, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.125@KRBTEST.COM for kudu/127.31.12.125@KRBTEST.COM
I20260924 15:22:53.990666 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:53.983527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:37555 (local address 127.0.0.1:45510)
0924 15:22:53.983723 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:53.983733 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:53.983786 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:53.983941 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:53.983942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:53.984094 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:53.984177 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:53.984179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:53.985072 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:53.985073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:53.985743 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:53.985746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:53.986290 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:53.986290 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:53.986342 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:53.986757 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:53.986761 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:53.987958 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:53.989638 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:53.989639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:53.989640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:53.989798 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:53.990027 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:53.990028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:53.990028 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:53.990082 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:53.990366 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:53.990367 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:53.990404 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 15:22:53.990465 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 15:22:53.990564 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181}
I20260924 15:22:53.990761 22076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:53.983600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45510 (local address 127.31.12.125:37555)
0924 15:22:53.983737 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:53.983740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:53.983780 (+ 40us) server_negotiation.cc:407] Connection header received
0924 15:22:53.983842 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:53.983843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:53.983853 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:53.983912 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:53.984258 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:53.984915 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:53.985818 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:53.986223 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:53.986280 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:53.988068 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:53.988076 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:53.988076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:53.988083 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:53.989552 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:53.989887 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:53.989890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:53.989890 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:53.989945 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:53.990151 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:53.990153 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:53.990153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:53.990291 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:53.990377 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:53.990492 (+ 115us) server_negotiation.cc:299] Negotiation successful
0924 15:22:53.990635 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
I20260924 15:22:53.991273 22072 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:22:53.991604 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:53.991508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:39671 (local address 127.0.0.1:56728)
0924 15:22:53.991547 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:53.991585 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 15:22:53.991782 22019 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.12.124:39671: connect: Connection refused (error 111)
I20260924 15:22:53.992187 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:53.992127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:39671 (local address 127.0.0.1:56732)
0924 15:22:53.992151 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:53.992171 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260924 15:22:53.992303 22072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.124:39671: Network error: Client connection negotiation failed: client connection to 127.31.12.124:39671: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.124.keytab.
Entry for principal kudu/127.31.12.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.124.keytab.
I20260924 15:22:54.001921 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.124:39671
--webserver_interface=127.31.12.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.124.keytab
--principal=kudu/127.31.12.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:38383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:35957,127.31.12.125:37555,127.31.12.124:39671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:22:54.035418 21995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } attempt: 3
Sep 24 15:22:54 dist-test-slave-zwbb krb5kdc[21912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263373, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for kudu/127.31.12.125@KRBTEST.COM
I20260924 15:22:54.041527 22072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } attempt: 1
I20260924 15:22:54.041874 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.041768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:39671 (local address 127.0.0.1:56738)
0924 15:22:54.041809 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:54.041837 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 15:22:54.042495 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.042385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:39671 (local address 127.0.0.1:56754)
0924 15:22:54.042438 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:54.042465 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 15:22:54.042646 22072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.124:39671: Network error: Client connection negotiation failed: client connection to 127.31.12.124:39671: connect: Connection refused (error 111)
I20260924 15:22:54.043207 22076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.035916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.31.12.125:37555)
0924 15:22:54.035989 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.035992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.036026 (+ 34us) server_negotiation.cc:407] Connection header received
0924 15:22:54.036061 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.036062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.036075 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.036128 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.036429 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.037243 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.038014 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.038379 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.038465 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:54.040480 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:54.040482 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:54.040482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:54.040490 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:54.042033 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.042339 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.042340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.042340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.042398 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.042648 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.042649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.042650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.042810 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:54.042898 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:54.043056 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 15:22:54.043127 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 15:22:54.043215 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.035846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:37555 (local address 127.0.0.1:45514)
0924 15:22:54.035936 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:54.035949 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:54.036014 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:54.036144 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:54.036145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:54.036290 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:54.036355 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:54.036356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.037329 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.037329 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:54.037906 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:54.037909 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.038470 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.038471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:54.038525 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:54.038978 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:54.038982 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:54.040336 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:54.042102 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:54.042103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:54.042103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:54.042272 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:54.042465 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:54.042466 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:54.042466 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:54.042543 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:54.042902 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:54.042903 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:54.042947 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 15:22:54.043031 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 15:22:54.043125 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260924 15:22:54.043932 21995 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:22:54.044261 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.044163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:39671 (local address 127.0.0.1:56770)
0924 15:22:54.044212 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:54.044237 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 15:22:54.044576 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.044522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:39671 (local address 127.0.0.1:56782)
0924 15:22:54.044546 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:54.044564 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260924 15:22:54.044672 21995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.124:39671: Network error: Client connection negotiation failed: client connection to 127.31.12.124:39671: connect: Connection refused (error 111)
I20260924 15:22:54.075801 21995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } attempt: 1
I20260924 15:22:54.076277 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.076118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:39671 (local address 127.0.0.1:56792)
0924 15:22:54.076191 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:54.076239 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260924 15:22:54.076702 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.076593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:39671 (local address 127.0.0.1:56798)
0924 15:22:54.076645 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:54.076677 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 15:22:54.076810 21995 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.124:39671: Network error: Client connection negotiation failed: client connection to 127.31.12.124:39671: connect: Connection refused (error 111)
W20260924 15:22:54.083758 22079 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 15:22:54.083938 22079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:54.083956 22079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:54.083976 22079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:54.085023 22079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:54.085057 22079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:54.085071 22079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:54.085084 22079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:54.085096 22079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:54.086613 22079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:38383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:35957,127.31.12.125:37555,127.31.12.124:39671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.124.keytab
--principal=kudu/127.31.12.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.124:39671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.12.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:54.086800 22079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:54.086949 22079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:54.091557 22087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:54.091557 22084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:54.091538 22085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:54.091842 22079 server_base.cc:1034] running on GCE node
I20260924 15:22:54.092095 22079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:54.092356 22079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:54.093523 22079 hybrid_clock.cc:648] HybridClock initialized: now 1790263374093499 us; error 24 us; skew 500 ppm
Sep 24 15:22:54 dist-test-slave-zwbb krb5kdc[21912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263374, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:54.095856 22079 init.cc:377] Logged in from keytab as kudu/127.31.12.124@KRBTEST.COM (short username kudu)
I20260924 15:22:54.096221 22079 webserver.cc:460] Webserver started at http://127.31.12.124:45257/ using document root <none> and password file <none>
I20260924 15:22:54.096415 22079 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:54.096457 22079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:54.096541 22079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:54.097620 22079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-2/data/instance:
uuid: "d652b3d607b34c8f9e511ac119b64829"
format_stamp: "Formatted at 2026-09-24 15:22:54 on dist-test-slave-zwbb"
server_key: "83fc902d7708ef02378ef1662f8daade"
server_key_iv: "78a5739dd1c4886f83402004e822b4a3"
server_key_version: "encryptionkey@0"
I20260924 15:22:54.097924 22079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-2/wal/instance:
uuid: "d652b3d607b34c8f9e511ac119b64829"
format_stamp: "Formatted at 2026-09-24 15:22:54 on dist-test-slave-zwbb"
server_key: "83fc902d7708ef02378ef1662f8daade"
server_key_iv: "78a5739dd1c4886f83402004e822b4a3"
server_key_version: "encryptionkey@0"
I20260924 15:22:54.099247 22079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:22:54.099938 22094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:54.100126 22079 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:22:54.100210 22079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-2/wal
uuid: "d652b3d607b34c8f9e511ac119b64829"
format_stamp: "Formatted at 2026-09-24 15:22:54 on dist-test-slave-zwbb"
server_key: "83fc902d7708ef02378ef1662f8daade"
server_key_iv: "78a5739dd1c4886f83402004e822b4a3"
server_key_version: "encryptionkey@0"
I20260924 15:22:54.100263 22079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:54.110803 22072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } attempt: 2
I20260924 15:22:54.111199 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.111076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:39671 (local address 127.0.0.1:56802)
0924 15:22:54.111136 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:54.111172 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 15:22:54.111577 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.111486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:39671 (local address 127.0.0.1:56818)
0924 15:22:54.111534 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:54.111559 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:39671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 15:22:54.111685 22072 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.124:39671: Network error: Client connection negotiation failed: client connection to 127.31.12.124:39671: connect: Connection refused (error 111)
I20260924 15:22:54.113061 22079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:54.114837 22079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:54.114943 22079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:54.118727 22079 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.124:39671
I20260924 15:22:54.118794 22148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.124:39671 every 8 connection(s)
I20260924 15:22:54.119115 22079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-2/data/info.pb
I20260924 15:22:54.120072 22149 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:54.121598 22149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:22:54.126345 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 22079
I20260924 15:22:54.126423 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790263080303576-31793-0/minicluster-data/master-2/wal/instance
I20260924 15:22:54.126626 31793 external_mini_cluster.cc:1428] Setting key a9d6ba075d22c5281da4db4c05a780f4
Sep 24 15:22:54 dist-test-slave-zwbb krb5kdc[21912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263374, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.124@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:22:54.130748 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.122210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:35957 (local address 127.0.0.1:40244)
0924 15:22:54.122405 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:54.122440 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:54.122937 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:54.123194 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:54.123200 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:54.123429 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:54.123605 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:54.123609 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.124253 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.124254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:54.125071 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:54.125073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.125497 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.125498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:54.125624 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:54.126105 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:54.126113 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:54.127721 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:54.129358 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:54.129365 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:54.129367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:54.129656 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:54.129865 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:54.129866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:54.129866 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:54.129985 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:54.130298 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:54.130304 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:54.130421 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 15:22:54.130503 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 15:22:54.130608 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":92,"threads_started":1}
I20260924 15:22:54.130750 21998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.122241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.31.12.126:35957)
0924 15:22:54.122300 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.122302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.122521 (+ 219us) server_negotiation.cc:407] Connection header received
0924 15:22:54.123042 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.123043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.123055 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.123116 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.123703 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.124158 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.125178 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.125370 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.125478 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:54.127817 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:54.127819 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:54.127820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:54.127827 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:54.129258 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.129740 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.129741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.129742 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.129800 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.130074 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.130075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.130075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.130224 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:54.130301 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:54.130552 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 15:22:54.130636 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 15:22:54.131636 22149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:22:54 dist-test-slave-zwbb krb5kdc[21912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.124@KRBTEST.COM
I20260924 15:22:54.135125 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.127030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56828 (local address 127.31.12.124:39671)
0924 15:22:54.127177 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.127180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.127344 (+ 164us) server_negotiation.cc:407] Connection header received
0924 15:22:54.127377 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.127380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.127406 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.127485 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.128052 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.128694 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.129416 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.129732 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.129812 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:54.132376 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:54.132384 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:54.132384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:54.132391 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:54.133915 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.134295 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.134298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.134298 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.134360 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.134572 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.134573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.134573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.134707 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:54.134851 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:54.134947 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 15:22:54.135022 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
Sep 24 15:22:54 dist-test-slave-zwbb krb5kdc[21912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263374, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.124@KRBTEST.COM for kudu/127.31.12.125@KRBTEST.COM
I20260924 15:22:54.138334 22076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.131960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45518 (local address 127.31.12.125:37555)
0924 15:22:54.132011 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.132013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.132024 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:22:54.132053 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.132054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.132062 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.132106 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.132372 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.133072 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.133751 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.133953 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.134027 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:54.135755 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:54.135756 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:54.135757 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:54.135763 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:54.137280 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.137685 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.137686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.137686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.137742 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.137913 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.137914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.137914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.138026 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:54.138092 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:54.138224 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 15:22:54.138278 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 15:22:54.138365 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.131873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:37555 (local address 127.0.0.1:45518)
0924 15:22:54.131947 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:54.131956 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:54.131990 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:54.132113 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:54.132114 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:54.132249 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:54.132305 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:54.132306 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.133145 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.133146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:54.133671 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:54.133673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.134026 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.134027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:54.134096 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:54.134479 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:54.134482 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:54.135667 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:54.137379 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:54.137379 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:54.137380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:54.137607 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:54.137807 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:54.137808 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:54.137808 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:54.137858 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:54.138085 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:54.138086 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:54.138142 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 15:22:54.138206 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 15:22:54.138282 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 15:22:54.139196 22149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:22:54 dist-test-slave-zwbb krb5kdc[21912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
Sep 24 15:22:54 dist-test-slave-zwbb krb5kdc[21912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263374, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.124@KRBTEST.COM for kudu/127.31.12.124@KRBTEST.COM
I20260924 15:22:54.143867 21998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.135936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40252 (local address 127.31.12.126:35957)
0924 15:22:54.135970 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.135972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.135980 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:22:54.136056 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.136057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.136063 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.136104 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.136670 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.137175 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.138128 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.138402 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.138464 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:54.140934 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:54.140936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:54.140936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:54.140943 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:54.142621 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.142982 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.142983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.142983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.143036 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.143239 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.143240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.143240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.143367 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:54.143514 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:54.143700 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 15:22:54.143775 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:22:54.145723 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.139492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56842 (local address 127.31.12.124:39671)
0924 15:22:54.139533 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.139535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.139582 (+ 47us) server_negotiation.cc:407] Connection header received
0924 15:22:54.139629 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.139629 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.139652 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.139700 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.139935 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.140483 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.141139 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.141345 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.141429 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:54.143097 (+ 1668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:54.143098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:54.143098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:54.143104 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:54.144571 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.144944 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.144945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.144945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.145004 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.145181 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.145182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.145182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.145298 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:54.145365 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:54.145526 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 15:22:54.145643 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":23}
I20260924 15:22:54.145726 22151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.139435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:39671 (local address 127.0.0.1:56842)
0924 15:22:54.139535 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:54.139551 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:54.139589 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:54.139713 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:54.139714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:54.139840 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:54.139866 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:54.139867 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.140560 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.140561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:54.141063 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:54.141064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.141430 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.141431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:54.141492 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:54.141804 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:54.141808 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:54.143023 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:54.144630 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:54.144631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:54.144631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:54.144853 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:54.145074 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:54.145074 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:54.145075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:54.145124 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:54.145361 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:54.145362 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:54.145400 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 15:22:54.145497 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 15:22:54.145600 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 15:22:54.147245 22149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829: Bootstrap starting.
I20260924 15:22:54.148067 22149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:54.148365 22149 log.cc:824] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829: Log is configured to *not* fsync() on all Append() calls
Sep 24 15:22:54 dist-test-slave-zwbb krb5kdc[21912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.125@KRBTEST.COM
I20260924 15:22:54.149197 22149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829: No bootstrap required, opened a new log
I20260924 15:22:54.151418 22149 raft_consensus.cc:348] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ccbeeac04a4006ab5326ea02841007" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } } peers { permanent_uuid: "6a8c5d13200847248f80cdbf527520ef" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } } peers { permanent_uuid: "d652b3d607b34c8f9e511ac119b64829" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } }
I20260924 15:22:54.151616 22149 raft_consensus.cc:729] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d652b3d607b34c8f9e511ac119b64829, State: Initialized, Role: FOLLOWER
I20260924 15:22:54.151707 22149 consensus_queue.cc:260] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ccbeeac04a4006ab5326ea02841007" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } } peers { permanent_uuid: "6a8c5d13200847248f80cdbf527520ef" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } } peers { permanent_uuid: "d652b3d607b34c8f9e511ac119b64829" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } }
I20260924 15:22:54.151926 21995 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } attempt: 2
I20260924 15:22:54.152094 22076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.144558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45522 (local address 127.31.12.125:37555)
0924 15:22:54.144591 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.144592 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.144600 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:22:54.144666 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.144667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.144675 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.144733 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.145373 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.146099 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.146780 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.147054 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.147130 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:54.149500 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:54.149507 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:54.149508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:54.149515 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:54.151016 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.151332 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.151333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.151334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.151394 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.151585 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.151586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.151587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.151711 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:54.151884 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:54.151926 (+ 42us) server_negotiation.cc:299] Negotiation successful
0924 15:22:54.151997 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 15:22:54.152627 22149 sys_catalog.cc:564] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:54.152930 22155 sys_catalog.cc:455] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ccbeeac04a4006ab5326ea02841007" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } } peers { permanent_uuid: "6a8c5d13200847248f80cdbf527520ef" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } } peers { permanent_uuid: "d652b3d607b34c8f9e511ac119b64829" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } } }
I20260924 15:22:54.153023 22155 sys_catalog.cc:458] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:22:54.153461 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 15:22:54.155175 22166 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:22:54.155283 22166 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:22:54.154495 21969 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "16ccbeeac04a4006ab5326ea02841007"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40252
Sep 24 15:22:54 dist-test-slave-zwbb krb5kdc[21912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263373, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for kudu/127.31.12.124@KRBTEST.COM
I20260924 15:22:54.155925 22079 master_runner.cc:186] Error getting master registration for 127.31.12.126:35957: OK, instance_id { permanent_uuid: "16ccbeeac04a4006ab5326ea02841007" instance_seqno: 1790263373810246 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 15:22:54.159017 22079 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 15:22:54.159456 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.152165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:39671 (local address 127.0.0.1:56858)
0924 15:22:54.152201 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:54.152212 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:54.152252 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:54.152430 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:54.152432 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:54.152621 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:54.152665 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:54.152667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.153472 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.153474 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:54.154216 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:54.154218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.154652 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.154653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:54.154716 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:54.155159 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:54.155164 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:54.156561 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:54.158381 (+ 1820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:54.158382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:54.158383 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:54.158604 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:54.158814 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:54.158815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:54.158815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:54.158873 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:54.159165 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:54.159166 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:54.159215 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 15:22:54.159285 (+ 70us) client_negotiation.cc:240] Negotiation successful
0924 15:22:54.159372 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 15:22:54.159483 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.152239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56858 (local address 127.31.12.124:39671)
0924 15:22:54.152279 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.152281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.152289 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:22:54.152307 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.152307 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.152317 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.152369 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.152746 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.153371 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.154303 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.154556 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.154635 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:54.156670 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:54.156671 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:54.156672 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:54.156678 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:54.158301 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.158682 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.158683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.158683 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.158742 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.158937 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.158938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.158938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.159090 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:54.159156 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:54.159318 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 15:22:54.159402 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:22:54.160881 21995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007: Bootstrap starting.
I20260924 15:22:54.161634 21995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:54.161914 21995 log.cc:824] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:54.162720 21995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007: No bootstrap required, opened a new log
I20260924 15:22:54.163641 21995 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ccbeeac04a4006ab5326ea02841007" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } } peers { permanent_uuid: "6a8c5d13200847248f80cdbf527520ef" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } } peers { permanent_uuid: "d652b3d607b34c8f9e511ac119b64829" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } }
I20260924 15:22:54.163765 21995 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ccbeeac04a4006ab5326ea02841007, State: Initialized, Role: FOLLOWER
I20260924 15:22:54.163851 21995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ccbeeac04a4006ab5326ea02841007" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } } peers { permanent_uuid: "6a8c5d13200847248f80cdbf527520ef" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } } peers { permanent_uuid: "d652b3d607b34c8f9e511ac119b64829" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } }
I20260924 15:22:54.164172 21995 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:54.164135 22169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ccbeeac04a4006ab5326ea02841007" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } } peers { permanent_uuid: "6a8c5d13200847248f80cdbf527520ef" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } } peers { permanent_uuid: "d652b3d607b34c8f9e511ac119b64829" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } } }
I20260924 15:22:54.164228 21969 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 15:22:54.164258 22169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:22:54.164306 21969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:54.165185 21969 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ccbeeac04a4006ab5326ea02841007" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } } peers { permanent_uuid: "6a8c5d13200847248f80cdbf527520ef" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } } peers { permanent_uuid: "d652b3d607b34c8f9e511ac119b64829" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } }
I20260924 15:22:54.165820 22123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16ccbeeac04a4006ab5326ea02841007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d652b3d607b34c8f9e511ac119b64829"
I20260924 15:22:54.165932 22123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:54.165709 21969 leader_election.cc:288] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [CANDIDATE]: Term 1 election: Requested vote from peers 6a8c5d13200847248f80cdbf527520ef (127.31.12.125:37555), d652b3d607b34c8f9e511ac119b64829 (127.31.12.124:39671)
I20260924 15:22:54.166882 22123 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16ccbeeac04a4006ab5326ea02841007 in term 1.
I20260924 15:22:54.167425 21943 leader_election.cc:302] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16ccbeeac04a4006ab5326ea02841007, d652b3d607b34c8f9e511ac119b64829; no voters:
I20260924 15:22:54.167567 22169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:54.167701 22169 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [term 1 LEADER]: Becoming Leader. State: Replica: 16ccbeeac04a4006ab5326ea02841007, State: Running, Role: LEADER
I20260924 15:22:54.167793 22169 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ccbeeac04a4006ab5326ea02841007" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } } peers { permanent_uuid: "6a8c5d13200847248f80cdbf527520ef" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } } peers { permanent_uuid: "d652b3d607b34c8f9e511ac119b64829" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } }
I20260924 15:22:54.168215 22169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16ccbeeac04a4006ab5326ea02841007. Latest consensus state: current_term: 1 leader_uuid: "16ccbeeac04a4006ab5326ea02841007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ccbeeac04a4006ab5326ea02841007" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } } peers { permanent_uuid: "6a8c5d13200847248f80cdbf527520ef" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } } peers { permanent_uuid: "d652b3d607b34c8f9e511ac119b64829" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } } }
I20260924 15:22:54.168309 22169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:54.166869 21926 master_runner.cc:186] Error getting master registration for 127.31.12.125:37555: OK, instance_id { permanent_uuid: "6a8c5d13200847248f80cdbf527520ef" instance_seqno: 1790263373968951 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 15:22:54.168926 21926 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 15:22:54.169605 22189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:54.170039 22189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:54.177022 22076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.169003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.31.12.125:37555)
0924 15:22:54.169052 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.169086 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.169137 (+ 51us) server_negotiation.cc:407] Connection header received
0924 15:22:54.169200 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.169201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.169211 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.169271 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.170051 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.170736 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.171779 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.172048 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.172093 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:54.173649 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:54.173651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:54.173652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:54.173659 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:54.175242 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.175949 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.175950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.175951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.176008 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.176305 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.176306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.176306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.176457 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:54.176530 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:54.176852 (+ 322us) server_negotiation.cc:299] Negotiation successful
0924 15:22:54.176932 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 15:22:54.178596 22123 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829 [term 1 FOLLOWER]: Refusing update from remote peer 16ccbeeac04a4006ab5326ea02841007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:22:54.178985 22169 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d652b3d607b34c8f9e511ac119b64829" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:22:54.179253 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.167263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.31.12.124:39671)
0924 15:22:54.167298 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.167300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.169274 (+ 1974us) server_negotiation.cc:407] Connection header received
0924 15:22:54.170782 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.170783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.170791 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.170819 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.171578 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.172242 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.173194 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.173499 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.173544 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:54.176440 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:54.176442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:54.176442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:54.176448 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:54.177997 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.178401 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.178402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.178403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.178458 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.178681 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.178682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.178682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.178801 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:54.178866 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:54.179074 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 15:22:54.179149 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 15:22:54.180228 22155 sys_catalog.cc:455] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16ccbeeac04a4006ab5326ea02841007. Latest consensus state: current_term: 1 leader_uuid: "16ccbeeac04a4006ab5326ea02841007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ccbeeac04a4006ab5326ea02841007" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } } peers { permanent_uuid: "6a8c5d13200847248f80cdbf527520ef" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } } peers { permanent_uuid: "d652b3d607b34c8f9e511ac119b64829" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } } }
I20260924 15:22:54.180310 22155 sys_catalog.cc:458] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:22:54.181962 22185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16ccbeeac04a4006ab5326ea02841007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ccbeeac04a4006ab5326ea02841007" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } } peers { permanent_uuid: "6a8c5d13200847248f80cdbf527520ef" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } } peers { permanent_uuid: "d652b3d607b34c8f9e511ac119b64829" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } } }
I20260924 15:22:54.182057 22185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:54.182514 22155 sys_catalog.cc:455] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16ccbeeac04a4006ab5326ea02841007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ccbeeac04a4006ab5326ea02841007" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } } peers { permanent_uuid: "6a8c5d13200847248f80cdbf527520ef" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } } peers { permanent_uuid: "d652b3d607b34c8f9e511ac119b64829" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } } }
I20260924 15:22:54.182595 22155 sys_catalog.cc:458] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:22:54.182895 22191 mvcc.cc:205] Tried to move back new op lower bound from 7332918780631965696 to 7332918780592070656. Current Snapshot: MvccSnapshot[applied={T|T < 7332918780631965696}]
I20260924 15:22:54.183063 22193 mvcc.cc:205] Tried to move back new op lower bound from 7332918780631965696 to 7332918780592070656. Current Snapshot: MvccSnapshot[applied={T|T < 7332918780631965696}]
I20260924 15:22:54.184243 22189 catalog_manager.cc:1347] Generated new cluster ID: 14c23a1d53b049bd857c689ebbf665d9
I20260924 15:22:54.184294 22189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:54.185338 21998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.169479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.31.12.126:35957)
0924 15:22:54.169522 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.169524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.177819 (+ 8295us) server_negotiation.cc:407] Connection header received
0924 15:22:54.177949 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.177950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.177969 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.177996 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.178844 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.179580 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.180609 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.180922 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.180965 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:54.182458 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:54.182459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:54.182460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:54.182467 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:54.184097 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.184497 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.184498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.184499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.184554 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.184745 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.184746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.184746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.184882 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:54.184957 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:54.185175 (+ 218us) server_negotiation.cc:299] Negotiation successful
0924 15:22:54.185251 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:22:54.193189 22189 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:54.194226 22189 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:54.202821 22072 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } attempt: 3
Sep 24 15:22:54 dist-test-slave-zwbb krb5kdc[21912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263373, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.125@KRBTEST.COM for kudu/127.31.12.124@KRBTEST.COM
I20260924 15:22:54.211838 22074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.203050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:39671 (local address 127.0.0.1:56872)
0924 15:22:54.203125 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:54.203135 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:54.203169 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:54.203520 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:54.203521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:54.203705 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:54.203749 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:54.203750 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.204685 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.204686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:54.205539 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:54.205542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.206719 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.206719 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:54.206779 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:54.207222 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:54.207227 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:54.208752 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:54.210668 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:54.210669 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:54.210670 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:54.210889 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:54.211141 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:54.211142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:54.211142 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:54.211211 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:54.211530 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:54.211531 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:54.211585 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 15:22:54.211646 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 15:22:54.211740 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 15:22:54.213035 22153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:54.203304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56872 (local address 127.31.12.124:39671)
0924 15:22:54.203375 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.203376 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.203383 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:22:54.203410 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.203411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.203420 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.203467 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.203837 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.204576 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.205631 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:54.205944 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:54.205999 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:54.208839 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:54.208840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:54.208841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:54.208847 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:54.210561 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.210995 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.210996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.210997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.211064 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:54.211284 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:54.211285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:54.211285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:54.211430 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:54.211489 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:54.212864 (+ 1375us) server_negotiation.cc:299] Negotiation successful
0924 15:22:54.212950 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 15:22:54.214812 22072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef: Bootstrap starting.
I20260924 15:22:54.215507 22072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:54.215809 22072 log.cc:824] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:54.217031 22189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007: Generated new TSK 0
I20260924 15:22:54.217283 22189 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 15:22:54.222950 22072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef: No bootstrap required, opened a new log
I20260924 15:22:54.224896 22072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ccbeeac04a4006ab5326ea02841007" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } } peers { permanent_uuid: "6a8c5d13200847248f80cdbf527520ef" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } } peers { permanent_uuid: "d652b3d607b34c8f9e511ac119b64829" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } }
I20260924 15:22:54.225055 22072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a8c5d13200847248f80cdbf527520ef, State: Initialized, Role: FOLLOWER
I20260924 15:22:54.225139 22072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ccbeeac04a4006ab5326ea02841007" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } } peers { permanent_uuid: "6a8c5d13200847248f80cdbf527520ef" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } } peers { permanent_uuid: "d652b3d607b34c8f9e511ac119b64829" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } }
I20260924 15:22:54.225509 22072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:54.225545 22046 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "16ccbeeac04a4006ab5326ea02841007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6a8c5d13200847248f80cdbf527520ef"
I20260924 15:22:54.225644 22046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:54.226485 22046 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16ccbeeac04a4006ab5326ea02841007 in term 1.
I20260924 15:22:54.225466 22198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ccbeeac04a4006ab5326ea02841007" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } } peers { permanent_uuid: "6a8c5d13200847248f80cdbf527520ef" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } } peers { permanent_uuid: "d652b3d607b34c8f9e511ac119b64829" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } } }
I20260924 15:22:54.226861 22045 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef [term 1 FOLLOWER]: Refusing update from remote peer 16ccbeeac04a4006ab5326ea02841007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:22:54.226861 22198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:22:54.227133 22169 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a8c5d13200847248f80cdbf527520ef" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:22:54.229713 22198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16ccbeeac04a4006ab5326ea02841007. Latest consensus state: current_term: 1 leader_uuid: "16ccbeeac04a4006ab5326ea02841007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ccbeeac04a4006ab5326ea02841007" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } } peers { permanent_uuid: "6a8c5d13200847248f80cdbf527520ef" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } } peers { permanent_uuid: "d652b3d607b34c8f9e511ac119b64829" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } } }
I20260924 15:22:54.229836 22198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:22:54.229907 22198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "16ccbeeac04a4006ab5326ea02841007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ccbeeac04a4006ab5326ea02841007" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } } peers { permanent_uuid: "6a8c5d13200847248f80cdbf527520ef" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } } peers { permanent_uuid: "d652b3d607b34c8f9e511ac119b64829" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } } }
I20260924 15:22:54.229990 22198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef [sys.catalog]: This master's current role is: FOLLOWER
W20260924 15:22:54.230928 22209 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:22:54.230985 22209 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:22:54.231236 22210 mvcc.cc:205] Tried to move back new op lower bound from 7332918780786692096 to 7332918780592070656. Current Snapshot: MvccSnapshot[applied={T|T < 7332918780631965696}]
W20260924 15:22:54.251685 22076 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:54.250438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.31.12.125:37555)
0924 15:22:54.250477 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.250479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.251046 (+ 567us) server_negotiation.cc:407] Connection header received
0924 15:22:54.251160 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.251161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.251172 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.251204 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.251630 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45536: BlockingRecv error: recv got EOF from 127.0.0.1:45536 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 15:22:54.252358 22153 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:54.250797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56888 (local address 127.31.12.124:39671)
0924 15:22:54.250829 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.250831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.251897 (+ 1066us) server_negotiation.cc:407] Connection header received
0924 15:22:54.251991 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.251992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.252002 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.252029 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.252303 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56888: BlockingRecv error: recv got EOF from 127.0.0.1:56888 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 15:22:54.253405 21998 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:54.252691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40264 (local address 127.31.12.126:35957)
0924 15:22:54.252727 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.252729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.252835 (+ 106us) server_negotiation.cc:407] Connection header received
0924 15:22:54.252919 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.252920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.252929 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.252956 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.253326 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40264: BlockingRecv error: recv got EOF from 127.0.0.1:40264 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 15:22:54.255458 22076 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:54.254817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45548 (local address 127.31.12.125:37555)
0924 15:22:54.254854 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.254855 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.254947 (+ 92us) server_negotiation.cc:407] Connection header received
0924 15:22:54.255050 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.255051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.255068 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.255095 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.255418 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45548: BlockingRecv error: recv got EOF from 127.0.0.1:45548 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260924 15:22:54.258302 22076 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:54.257606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.31.12.125:37555)
0924 15:22:54.257637 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.257639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.257859 (+ 220us) server_negotiation.cc:407] Connection header received
0924 15:22:54.257946 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.257946 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.257955 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.257982 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.258265 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45558: BlockingRecv error: recv got EOF from 127.0.0.1:45558 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 15:22:54.262204 22076 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:54.261536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45568 (local address 127.31.12.125:37555)
0924 15:22:54.261570 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.261572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.261791 (+ 219us) server_negotiation.cc:407] Connection header received
0924 15:22:54.261874 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.261875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.261883 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.261908 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.262167 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45568: BlockingRecv error: recv got EOF from 127.0.0.1:45568 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 15:22:54.267889 22076 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:54.267144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45584 (local address 127.31.12.125:37555)
0924 15:22:54.267175 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.267177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.267413 (+ 236us) server_negotiation.cc:407] Connection header received
0924 15:22:54.267504 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.267504 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.267513 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.267540 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.267852 (+ 312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45584: BlockingRecv error: recv got EOF from 127.0.0.1:45584 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 15:22:54.277750 22076 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:54.276899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45600 (local address 127.31.12.125:37555)
0924 15:22:54.276945 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.276947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.277176 (+ 229us) server_negotiation.cc:407] Connection header received
0924 15:22:54.277270 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.277270 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.277280 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.277307 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.277694 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45600: BlockingRecv error: recv got EOF from 127.0.0.1:45600 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260924 15:22:54.295697 22076 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:54.294831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45612 (local address 127.31.12.125:37555)
0924 15:22:54.294869 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.294871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.295143 (+ 272us) server_negotiation.cc:407] Connection header received
0924 15:22:54.295237 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.295238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.295248 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.295276 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.295652 (+ 376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45612: BlockingRecv error: recv got EOF from 127.0.0.1:45612 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 15:22:54.329924 22076 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:54.328927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45624 (local address 127.31.12.125:37555)
0924 15:22:54.328970 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.328972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.329267 (+ 295us) server_negotiation.cc:407] Connection header received
0924 15:22:54.329370 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.329371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.329382 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.329427 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.329867 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45624: BlockingRecv error: recv got EOF from 127.0.0.1:45624 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 15:22:54.395970 22076 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:54.395207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45628 (local address 127.31.12.125:37555)
0924 15:22:54.395272 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.395274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.395396 (+ 122us) server_negotiation.cc:407] Connection header received
0924 15:22:54.395476 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.395477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.395487 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.395549 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.395916 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45628: BlockingRecv error: recv got EOF from 127.0.0.1:45628 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 15:22:54.526149 22076 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:54.525420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45632 (local address 127.31.12.125:37555)
0924 15:22:54.525541 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.525544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.525641 (+ 97us) server_negotiation.cc:407] Connection header received
0924 15:22:54.525673 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.525674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.525686 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.525750 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.526086 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45632: BlockingRecv error: recv got EOF from 127.0.0.1:45632 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 15:22:54.735229 22272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "16ccbeeac04a4006ab5326ea02841007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16ccbeeac04a4006ab5326ea02841007" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 35957 } } peers { permanent_uuid: "6a8c5d13200847248f80cdbf527520ef" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 37555 } } peers { permanent_uuid: "d652b3d607b34c8f9e511ac119b64829" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 39671 } } }
I20260924 15:22:54.735394 22272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ccbeeac04a4006ab5326ea02841007 [sys.catalog]: This master's current role is: LEADER
W20260924 15:22:54.784032 22076 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:54.783304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45638 (local address 127.31.12.125:37555)
0924 15:22:54.783389 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:54.783391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:54.783406 (+ 15us) server_negotiation.cc:407] Connection header received
0924 15:22:54.783520 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:54.783521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:54.783533 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:54.783613 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:54.783968 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45638: BlockingRecv error: recv got EOF from 127.0.0.1:45638 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 15:22:55.155788 22166 catalog_manager.cc:1259] Loaded cluster ID: 14c23a1d53b049bd857c689ebbf665d9
I20260924 15:22:55.155867 22166 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829: loading cluster ID for follower catalog manager: success
I20260924 15:22:55.156674 22166 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829: acquiring CA information for follower catalog manager: success
I20260924 15:22:55.157014 22166 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d652b3d607b34c8f9e511ac119b64829: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:22:55.231410 22209 catalog_manager.cc:1259] Loaded cluster ID: 14c23a1d53b049bd857c689ebbf665d9
I20260924 15:22:55.231477 22209 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef: loading cluster ID for follower catalog manager: success
I20260924 15:22:55.232300 22209 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef: acquiring CA information for follower catalog manager: success
I20260924 15:22:55.232628 22209 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6a8c5d13200847248f80cdbf527520ef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:22:55.299870 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:55.297151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45644 (local address 127.31.12.125:37555)
0924 15:22:55.297273 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:55.297276 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:55.297349 (+ 73us) server_negotiation.cc:407] Connection header received
0924 15:22:55.297478 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:55.297479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:55.297546 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:55.297634 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:22:55.297782 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:55.298271 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:55.299014 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:55.299255 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:55.299334 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:55.299509 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:22:55.299629 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:22:55.299708 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:55.299757 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 15:22:55.299777 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260924 15:22:55.304039 22290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:55.301762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56890 (local address 127.31.12.124:39671)
0924 15:22:55.301864 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:55.301866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:55.301918 (+ 52us) server_negotiation.cc:407] Connection header received
0924 15:22:55.301987 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:55.301988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:55.302023 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:55.302099 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:22:55.302175 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:55.302624 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:55.303371 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:55.303631 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:55.303677 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:55.303794 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:22:55.303904 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:22:55.303955 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:55.303968 (+ 13us) server_negotiation.cc:299] Negotiation successful
0924 15:22:55.303982 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":50,"threads_started":1}
I20260924 15:22:55.308808 22290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:55.305833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56900 (local address 127.31.12.124:39671)
0924 15:22:55.305872 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:55.305874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:55.305963 (+ 89us) server_negotiation.cc:407] Connection header received
0924 15:22:55.306009 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:55.306009 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:55.306017 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:55.306088 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:22:55.306350 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:55.306890 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:55.307935 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:55.308210 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:55.308273 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:55.308430 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:22:55.308462 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:22:55.308507 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:55.308669 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 15:22:55.308689 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 15:22:55.308920 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:55.305732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45652 (local address 127.31.12.125:37555)
0924 15:22:55.305778 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:55.305781 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:55.306085 (+ 304us) server_negotiation.cc:407] Connection header received
0924 15:22:55.306137 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:55.306138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:55.306152 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:55.306210 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:22:55.306342 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:55.307060 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:55.308118 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:55.308402 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:55.308478 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:55.308643 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:22:55.308679 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:22:55.308746 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:55.308814 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 15:22:55.308833 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:22:55.309520 22298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:55.305786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40278 (local address 127.31.12.126:35957)
0924 15:22:55.305893 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:55.305895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:55.306267 (+ 372us) server_negotiation.cc:407] Connection header received
0924 15:22:55.306345 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:55.306346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:55.306394 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:55.306473 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:22:55.306621 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:55.307318 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:55.308180 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:55.308502 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:55.308561 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:55.308843 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:22:55.309117 (+ 274us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:22:55.309179 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:55.309268 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 15:22:55.309298 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":49,"threads_started":1}
I20260924 15:22:55.311369 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 21926
I20260924 15:22:55.317020 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 22002
I20260924 15:22:55.321878 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 22079
2026-09-24T15:22:55Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2761 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 15:22:55 dist-test-slave-zwbb krb5kdc[22301](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 15:22:55 dist-test-slave-zwbb krb5kdc[22301](info): set up 2 sockets
Sep 24 15:22:55 dist-test-slave-zwbb krb5kdc[22301](info): commencing operation
krb5kdc: starting...
W20260924 15:22:56.343233 31793 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/test-user.keytab.
Sep 24 15:22:56 dist-test-slave-zwbb krb5kdc[22301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T15:22:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T15:22:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.31.12.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
Entry for principal kudu/127.31.12.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab.
I20260924 15:22:56.458212 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:38605
--webserver_interface=127.31.12.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:41103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:38605,127.31.12.125:34071,127.31.12.124:35095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 15:22:56.535248 22314 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 15:22:56.535460 22314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:56.535478 22314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:56.535499 22314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:56.536530 22314 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:56.536562 22314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:56.536576 22314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:56.536588 22314 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:56.536600 22314 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:56.538030 22314 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:41103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:38605,127.31.12.125:34071,127.31.12.124:35095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.126.keytab
--principal=kudu/127.31.12.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.126:38605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.31.12.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:56.538208 22314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:56.538362 22314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:56.543002 22320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:56.543010 22319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:56.543010 22322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:56.543339 22314 server_base.cc:1034] running on GCE node
I20260924 15:22:56.543541 22314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:56.543800 22314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:56.544951 22314 hybrid_clock.cc:648] HybridClock initialized: now 1790263376544941 us; error 27 us; skew 500 ppm
Sep 24 15:22:56 dist-test-slave-zwbb krb5kdc[22301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263376, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:56.550694 22314 init.cc:377] Logged in from keytab as kudu/127.31.12.126@KRBTEST.COM (short username kudu)
I20260924 15:22:56.551184 22314 webserver.cc:460] Webserver started at http://127.31.12.126:36077/ using document root <none> and password file <none>
I20260924 15:22:56.551421 22314 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:56.551491 22314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:56.551641 22314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:56.552670 22314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-0/data/instance:
uuid: "71e972e31250440ea7b6c42bf8fd2320"
format_stamp: "Formatted at 2026-09-24 15:22:56 on dist-test-slave-zwbb"
server_key: "c69b557c2a3ca79953f8b4319de45da8"
server_key_iv: "41134eea313201bbe1a42ba8d9278f01"
server_key_version: "encryptionkey@0"
I20260924 15:22:56.553043 22314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-0/wal/instance:
uuid: "71e972e31250440ea7b6c42bf8fd2320"
format_stamp: "Formatted at 2026-09-24 15:22:56 on dist-test-slave-zwbb"
server_key: "c69b557c2a3ca79953f8b4319de45da8"
server_key_iv: "41134eea313201bbe1a42ba8d9278f01"
server_key_version: "encryptionkey@0"
I20260924 15:22:56.554430 22314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 15:22:56.555187 22329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:56.555357 22314 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:22:56.555470 22314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-0/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-0/wal
uuid: "71e972e31250440ea7b6c42bf8fd2320"
format_stamp: "Formatted at 2026-09-24 15:22:56 on dist-test-slave-zwbb"
server_key: "c69b557c2a3ca79953f8b4319de45da8"
server_key_iv: "41134eea313201bbe1a42ba8d9278f01"
server_key_version: "encryptionkey@0"
I20260924 15:22:56.555545 22314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:56.578895 22314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:56.580721 22314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:56.580892 22314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:56.585377 22314 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.126:38605
I20260924 15:22:56.585417 22381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.126:38605 every 8 connection(s)
I20260924 15:22:56.585803 22314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-0/data/info.pb
I20260924 15:22:56.586755 22382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:56.588294 22382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:22:56.592881 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 22314
I20260924 15:22:56.592972 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-0/wal/instance
I20260924 15:22:56.593215 31793 external_mini_cluster.cc:1428] Setting key ecb17f5600168db379d29e1bb7ce7782
Sep 24 15:22:56 dist-test-slave-zwbb krb5kdc[22301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263376, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:22:56.598162 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.589030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42136 (local address 127.31.12.126:38605)
0924 15:22:56.589274 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:56.589278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:56.589303 (+ 25us) server_negotiation.cc:407] Connection header received
0924 15:22:56.590021 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:56.590023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:56.590040 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:56.590122 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:56.590652 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.591437 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.592239 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.592515 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.592586 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.594878 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:56.594885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:56.594885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:56.594891 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:56.596408 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.596897 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.596899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.596900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.597049 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.597316 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.597317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.597317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.597603 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:56.597667 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:56.597919 (+ 252us) server_negotiation.cc:299] Negotiation successful
0924 15:22:56.598016 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":103,"threads_started":1}
I20260924 15:22:56.598275 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.589018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38605 (local address 127.0.0.1:42136)
0924 15:22:56.589199 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.589233 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:56.589871 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:56.590153 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:56.590156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:56.590407 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:56.590561 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:56.590565 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.591534 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.591535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:56.592171 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:56.592173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.592600 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.592601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:56.592686 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.593246 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:56.593256 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:56.594794 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:56.596498 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:56.596505 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:56.596507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:56.596794 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:56.597134 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:56.597134 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:56.597135 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:56.597249 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:56.597682 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:56.597690 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:56.597769 (+ 79us) client_negotiation.cc:769] Sending connection context
0924 15:22:56.597878 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 15:22:56.598100 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":58,"thread_start_us":100,"threads_started":1}
I20260924 15:22:56.599020 22382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:22:56.599372 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.599222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:34071 (local address 127.0.0.1:54702)
0924 15:22:56.599278 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.599328 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.125:34071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 15:22:56.599531 22333 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.12.125:34071: connect: Connection refused (error 111)
I20260924 15:22:56.600028 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.599919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:34071 (local address 127.0.0.1:54708)
0924 15:22:56.599970 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.600003 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.125:34071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 15:22:56.600154 22382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.125:34071: Network error: Client connection negotiation failed: client connection to 127.31.12.125:34071: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.31.12.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.125.keytab.
Entry for principal kudu/127.31.12.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.125.keytab.
I20260924 15:22:56.619831 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.125:34071
--webserver_interface=127.31.12.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.125.keytab
--principal=kudu/127.31.12.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:41103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:38605,127.31.12.125:34071,127.31.12.124:35095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:22:56.649402 22382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } attempt: 1
I20260924 15:22:56.649955 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.649821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:34071 (local address 127.0.0.1:54714)
0924 15:22:56.649887 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.649926 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.125:34071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260924 15:22:56.650332 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.650250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:34071 (local address 127.0.0.1:54724)
0924 15:22:56.650287 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.650309 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.125:34071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 15:22:56.650435 22382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.125:34071: Network error: Client connection negotiation failed: client connection to 127.31.12.125:34071: connect: Connection refused (error 111)
W20260924 15:22:56.698446 22389 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 15:22:56.698649 22389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:56.698684 22389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:56.698704 22389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:56.699746 22389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:56.699780 22389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:56.699795 22389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:56.699808 22389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:56.699820 22389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:56.701208 22389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:41103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:38605,127.31.12.125:34071,127.31.12.124:35095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.125.keytab
--principal=kudu/127.31.12.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.125:34071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.31.12.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:56.701385 22389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:56.701557 22389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 15:22:56.706459 22389 server_base.cc:1034] running on GCE node
W20260924 15:22:56.706579 22398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:56.706614 22395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:56.706729 22396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:56.707070 22389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:56.707414 22389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:56.708597 22389 hybrid_clock.cc:648] HybridClock initialized: now 1790263376708572 us; error 35 us; skew 500 ppm
Sep 24 15:22:56 dist-test-slave-zwbb krb5kdc[22301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263376, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:56.711891 22389 init.cc:377] Logged in from keytab as kudu/127.31.12.125@KRBTEST.COM (short username kudu)
I20260924 15:22:56.712416 22389 webserver.cc:460] Webserver started at http://127.31.12.125:44191/ using document root <none> and password file <none>
I20260924 15:22:56.712662 22389 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:56.712733 22389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:56.712882 22389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:56.714157 22389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-1/data/instance:
uuid: "0cd8c269cf3643669d5ca88a33f0bfe2"
format_stamp: "Formatted at 2026-09-24 15:22:56 on dist-test-slave-zwbb"
server_key: "0bffe2d99c243cd8e2b4ca263afa3ee8"
server_key_iv: "b63b310037b8cac440e269cd859ab2a4"
server_key_version: "encryptionkey@0"
I20260924 15:22:56.714485 22389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-1/wal/instance:
uuid: "0cd8c269cf3643669d5ca88a33f0bfe2"
format_stamp: "Formatted at 2026-09-24 15:22:56 on dist-test-slave-zwbb"
server_key: "0bffe2d99c243cd8e2b4ca263afa3ee8"
server_key_iv: "b63b310037b8cac440e269cd859ab2a4"
server_key_version: "encryptionkey@0"
I20260924 15:22:56.715934 22389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 15:22:56.716549 22405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:56.716701 22389 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 15:22:56.716786 22389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-1/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-1/wal
uuid: "0cd8c269cf3643669d5ca88a33f0bfe2"
format_stamp: "Formatted at 2026-09-24 15:22:56 on dist-test-slave-zwbb"
server_key: "0bffe2d99c243cd8e2b4ca263afa3ee8"
server_key_iv: "b63b310037b8cac440e269cd859ab2a4"
server_key_version: "encryptionkey@0"
I20260924 15:22:56.716847 22389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:56.718568 22382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } attempt: 2
I20260924 15:22:56.718950 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.718813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:34071 (local address 127.0.0.1:54732)
0924 15:22:56.718870 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.718900 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.125:34071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 15:22:56.719334 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.719253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:34071 (local address 127.0.0.1:54736)
0924 15:22:56.719297 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.719320 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.125:34071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260924 15:22:56.719442 22382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.125:34071: Network error: Client connection negotiation failed: client connection to 127.31.12.125:34071: connect: Connection refused (error 111)
I20260924 15:22:56.735955 22389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:56.737782 22389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:56.737880 22389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:56.741655 22389 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.125:34071
I20260924 15:22:56.741734 22458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.125:34071 every 8 connection(s)
I20260924 15:22:56.742051 22389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-1/data/info.pb
I20260924 15:22:56.742945 22459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:56.744433 22459 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:22:56.745267 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 22389
I20260924 15:22:56.745370 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-1/wal/instance
I20260924 15:22:56.745586 31793 external_mini_cluster.cc:1428] Setting key 21d5c8f3b60e16f2c89ee00c10d014c2
Sep 24 15:22:56 dist-test-slave-zwbb krb5kdc[22301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263376, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.125@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.31.12.124@KRBTEST.COM; defaulting to no policy
I20260924 15:22:56.753338 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.745102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42152 (local address 127.31.12.126:38605)
0924 15:22:56.745178 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:56.745181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:56.745379 (+ 198us) server_negotiation.cc:407] Connection header received
0924 15:22:56.745952 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:56.745953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:56.745968 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:56.746018 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:56.746616 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.747179 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.748251 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.748477 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.748557 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.750627 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:56.750629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:56.750630 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:56.750638 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:56.752020 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.752422 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.752423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.752424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.752478 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.752745 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.752746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.752746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.752853 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:56.752929 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:56.753175 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 15:22:56.753261 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 15:22:56.753575 22461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.745080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38605 (local address 127.0.0.1:42152)
0924 15:22:56.745239 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.745279 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:56.745849 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:56.746099 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:56.746102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:56.746330 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:56.746516 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:56.746526 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.747341 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.747343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:56.748155 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:56.748158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.748576 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.748578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:56.748714 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.749183 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:56.749191 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:56.750533 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:56.752112 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:56.752120 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:56.752122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:56.752333 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:56.752542 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:56.752543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:56.752544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:56.752692 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:56.752921 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:56.752928 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:56.753055 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 15:22:56.753155 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 15:22:56.753300 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260924 15:22:56.754113 22459 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.31.12.124@KRBTEST.COM" created.
Sep 24 15:22:56 dist-test-slave-zwbb krb5kdc[22301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263376, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.125@KRBTEST.COM for kudu/127.31.12.125@KRBTEST.COM
I20260924 15:22:56.761756 22461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.754395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:34071 (local address 127.0.0.1:54752)
0924 15:22:56.754440 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.754450 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:56.754485 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:56.754735 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:56.754736 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:56.754841 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:56.754885 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:56.754887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.755706 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.755707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:56.756586 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:56.756589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.757252 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.757253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:56.757304 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.757717 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:56.757721 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:56.758916 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:56.760645 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:56.760646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:56.760647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:56.760807 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:56.761042 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:56.761043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:56.761043 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:56.761096 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:56.761403 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:56.761403 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:56.761465 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 15:22:56.761537 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 15:22:56.761629 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 15:22:56.761929 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.754430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54752 (local address 127.31.12.125:34071)
0924 15:22:56.754602 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:56.754606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:56.754625 (+ 19us) server_negotiation.cc:407] Connection header received
0924 15:22:56.754645 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:56.754647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:56.754663 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:56.754721 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:56.754952 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.755623 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.756677 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.757173 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.757247 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.759006 (+ 1759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:56.759013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:56.759014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:56.759020 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:56.760546 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.760913 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.760916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.760916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.760971 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.761171 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.761172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.761172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.761306 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:56.761420 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:56.761569 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 15:22:56.761681 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":144,"thread_start_us":121,"threads_started":1}
I20260924 15:22:56.762444 22459 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:22:56.762758 22461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.762669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:35095 (local address 127.0.0.1:59118)
0924 15:22:56.762701 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.762739 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:35095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260924 15:22:56.762909 22408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.31.12.124:35095: connect: Connection refused (error 111)
I20260924 15:22:56.763294 22461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.763236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:35095 (local address 127.0.0.1:59128)
0924 15:22:56.763259 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.763280 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:35095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260924 15:22:56.763417 22459 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.124:35095: Network error: Client connection negotiation failed: client connection to 127.31.12.124:35095: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.31.12.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.124.keytab.
Entry for principal kudu/127.31.12.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.124.keytab.
I20260924 15:22:56.772006 31793 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
/tmp/dist-test-taskqOs6Q8/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.124:35095
--webserver_interface=127.31.12.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.124.keytab
--principal=kudu/127.31.12.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.31.12.84:41103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.31.12.126:38605,127.31.12.125:34071,127.31.12.124:35095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 15:22:56.810583 22382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } attempt: 3
I20260924 15:22:56.812654 22459 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } attempt: 1
I20260924 15:22:56.813059 22461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.812916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:35095 (local address 127.0.0.1:59142)
0924 15:22:56.812977 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.813017 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:35095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 15:22:56.813387 22461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.813290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:35095 (local address 127.0.0.1:59152)
0924 15:22:56.813322 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.813361 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:35095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260924 15:22:56.813519 22459 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.124:35095: Network error: Client connection negotiation failed: client connection to 127.31.12.124:35095: connect: Connection refused (error 111)
Sep 24 15:22:56 dist-test-slave-zwbb krb5kdc[22301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263376, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for kudu/127.31.12.125@KRBTEST.COM
I20260924 15:22:56.817936 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.811011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54764 (local address 127.31.12.125:34071)
0924 15:22:56.811080 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:56.811082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:56.811094 (+ 12us) server_negotiation.cc:407] Connection header received
0924 15:22:56.811133 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:56.811134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:56.811152 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:56.811209 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:56.811549 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.812049 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.812868 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.813200 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.813321 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.815347 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:56.815349 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:56.815349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:56.815357 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:56.816709 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.817074 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.817075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.817075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.817134 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.817370 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.817371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.817372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.817539 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:56.817608 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:56.817770 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 15:22:56.817856 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 15:22:56.817942 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.810934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:34071 (local address 127.0.0.1:54764)
0924 15:22:56.810998 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.811008 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:56.811069 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:56.811214 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:56.811215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:56.811401 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:56.811464 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:56.811466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.812145 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.812146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:56.812695 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:56.812698 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.813347 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.813348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:56.813467 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.813914 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:56.813920 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:56.815261 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:56.816804 (+ 1543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:56.816805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:56.816806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:56.816986 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:56.817225 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:56.817226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:56.817226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:56.817291 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:56.817612 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:56.817613 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:56.817660 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 15:22:56.817743 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 15:22:56.817847 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 15:22:56.818280 22382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:22:56.818616 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.818521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:35095 (local address 127.0.0.1:59162)
0924 15:22:56.818568 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.818594 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:35095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 15:22:56.818967 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.818904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:35095 (local address 127.0.0.1:59164)
0924 15:22:56.818936 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.818955 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:35095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260924 15:22:56.819078 22382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.124:35095: Network error: Client connection negotiation failed: client connection to 127.31.12.124:35095: connect: Connection refused (error 111)
I20260924 15:22:56.850210 22382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } attempt: 1
I20260924 15:22:56.850694 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.850548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:35095 (local address 127.0.0.1:59172)
0924 15:22:56.850633 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.850669 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:35095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 15:22:56.851044 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.850957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:35095 (local address 127.0.0.1:59188)
0924 15:22:56.850997 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.851027 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:35095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 15:22:56.851135 22382 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.124:35095: Network error: Client connection negotiation failed: client connection to 127.31.12.124:35095: connect: Connection refused (error 111)
W20260924 15:22:56.852248 22466 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 15:22:56.852444 22466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 15:22:56.852495 22466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 15:22:56.852535 22466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 15:22:56.853600 22466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 15:22:56.853652 22466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 15:22:56.853673 22466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 15:22:56.853699 22466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 15:22:56.853724 22466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 15:22:56.855129 22466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.31.12.84:41103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.31.12.126:38605,127.31.12.125:34071,127.31.12.124:35095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/krb5kdc/kudu_127.31.12.124.keytab
--principal=kudu/127.31.12.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.31.12.124:35095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.31.12.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 14:28:46 UTC on ccycloud.quasar-mbdzkc.root.comops.site
build id 4670395
I20260924 15:22:56.855373 22466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 15:22:56.855571 22466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 15:22:56.860002 22472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:56.860026 22474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 15:22:56.860036 22471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 15:22:56.860149 22466 server_base.cc:1034] running on GCE node
I20260924 15:22:56.860415 22466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 15:22:56.860651 22466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 15:22:56.861789 22466 hybrid_clock.cc:648] HybridClock initialized: now 1790263376861779 us; error 23 us; skew 500 ppm
Sep 24 15:22:56 dist-test-slave-zwbb krb5kdc[22301](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263376, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 15:22:56.864114 22466 init.cc:377] Logged in from keytab as kudu/127.31.12.124@KRBTEST.COM (short username kudu)
I20260924 15:22:56.864467 22466 webserver.cc:460] Webserver started at http://127.31.12.124:41535/ using document root <none> and password file <none>
I20260924 15:22:56.864632 22466 fs_manager.cc:359] Metadata directory not provided
I20260924 15:22:56.864670 22466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 15:22:56.864754 22466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 15:22:56.865779 22466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-2/data/instance:
uuid: "e7e92d7a380048d39aacecc1e19a599e"
format_stamp: "Formatted at 2026-09-24 15:22:56 on dist-test-slave-zwbb"
server_key: "368c1c3929dded15d321f1837d7d06d0"
server_key_iv: "1f86b67baf4932bd43546ed3cd6e7e63"
server_key_version: "encryptionkey@0"
I20260924 15:22:56.866073 22466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-2/wal/instance:
uuid: "e7e92d7a380048d39aacecc1e19a599e"
format_stamp: "Formatted at 2026-09-24 15:22:56 on dist-test-slave-zwbb"
server_key: "368c1c3929dded15d321f1837d7d06d0"
server_key_iv: "1f86b67baf4932bd43546ed3cd6e7e63"
server_key_version: "encryptionkey@0"
I20260924 15:22:56.867278 22466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 15:22:56.867913 22481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 15:22:56.868081 22466 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 15:22:56.868172 22466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-2/data,/tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-2/wal
uuid: "e7e92d7a380048d39aacecc1e19a599e"
format_stamp: "Formatted at 2026-09-24 15:22:56 on dist-test-slave-zwbb"
server_key: "368c1c3929dded15d321f1837d7d06d0"
server_key_iv: "1f86b67baf4932bd43546ed3cd6e7e63"
server_key_version: "encryptionkey@0"
I20260924 15:22:56.868247 22466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 15:22:56.881664 22459 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } attempt: 2
I20260924 15:22:56.882071 22461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.881952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:35095 (local address 127.0.0.1:59190)
0924 15:22:56.882006 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.882041 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:35095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260924 15:22:56.882400 22461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.882309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:35095 (local address 127.0.0.1:59200)
0924 15:22:56.882351 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.882376 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.31.12.124:35095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 15:22:56.882511 22459 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.31.12.124:35095: Network error: Client connection negotiation failed: client connection to 127.31.12.124:35095: connect: Connection refused (error 111)
I20260924 15:22:56.907207 22466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 15:22:56.909081 22466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 15:22:56.909188 22466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 15:22:56.913030 22466 rpc_server.cc:307] RPC server started. Bound to: 127.31.12.124:35095
I20260924 15:22:56.913077 22533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.31.12.124:35095 every 8 connection(s)
I20260924 15:22:56.913425 22466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-2/data/info.pb
I20260924 15:22:56.914513 22534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 15:22:56.915983 22534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:22:56.916347 31793 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu as pid 22466
I20260924 15:22:56.916457 31793 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskqOs6Q8/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790263080303576-31793-0/minicluster-data/master-2/wal/instance
I20260924 15:22:56.916697 31793 external_mini_cluster.cc:1428] Setting key 1ca6361303f7c73ff90bdba957572cfa
Sep 24 15:22:56 dist-test-slave-zwbb krb5kdc[22301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263376, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.124@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
Sep 24 15:22:56 dist-test-slave-zwbb krb5kdc[22301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.124@KRBTEST.COM
I20260924 15:22:56.925925 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.916762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42162 (local address 127.31.12.126:38605)
0924 15:22:56.916814 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:56.916816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:56.917105 (+ 289us) server_negotiation.cc:407] Connection header received
0924 15:22:56.917778 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:56.917779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:56.917793 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:56.917856 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:56.918382 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.918874 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.920006 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.920300 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.920405 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.922763 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:56.922764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:56.922765 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:56.922772 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:56.924362 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.924957 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.924958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.924958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.925009 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.925274 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.925274 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.925275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.925372 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:56.925423 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:56.925746 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 15:22:56.925827 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 15:22:56.926093 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.916739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.126:38605 (local address 127.0.0.1:42162)
0924 15:22:56.916950 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.916998 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:56.917655 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:56.917887 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:56.917889 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:56.918144 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:56.918289 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:56.918293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.918948 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.918949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:56.919928 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:56.919931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.920427 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.920428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:56.920571 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.921129 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:56.921138 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:56.922664 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:56.924460 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:56.924468 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:56.924470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:56.924880 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:56.925076 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:56.925077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:56.925077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:56.925199 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:56.925458 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:56.925464 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:56.925616 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 15:22:56.925696 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 15:22:56.925852 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":35,"thread_start_us":92,"threads_started":1}
I20260924 15:22:56.926384 22382 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } attempt: 2
I20260924 15:22:56.926626 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.917166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59206 (local address 127.31.12.124:35095)
0924 15:22:56.917307 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:56.917312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:56.917347 (+ 35us) server_negotiation.cc:407] Connection header received
0924 15:22:56.917657 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:56.917666 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:56.917693 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:56.917779 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:56.918458 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.919488 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.920538 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.920989 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.921105 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.923670 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:56.923677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:56.923677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:56.923684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:56.925441 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.925782 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.925785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.925786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.925841 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.926070 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.926071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.926071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.926196 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:56.926296 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:56.926450 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 15:22:56.926529 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260924 15:22:56.927062 22534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 15:22:56 dist-test-slave-zwbb krb5kdc[22301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263376, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.126@KRBTEST.COM for kudu/127.31.12.124@KRBTEST.COM
Sep 24 15:22:56 dist-test-slave-zwbb krb5kdc[22301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263376, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.124@KRBTEST.COM for kudu/127.31.12.125@KRBTEST.COM
Sep 24 15:22:56 dist-test-slave-zwbb krb5kdc[22301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.126@KRBTEST.COM
I20260924 15:22:56.934821 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.927490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54778 (local address 127.31.12.125:34071)
0924 15:22:56.927535 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:56.927537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:56.927695 (+ 158us) server_negotiation.cc:407] Connection header received
0924 15:22:56.927785 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:56.927786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:56.927795 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:56.927824 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:56.928150 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.928660 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.929529 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.929850 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.929951 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.931782 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:56.931783 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:56.931783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:56.931791 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:56.933493 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.933860 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.933861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.933862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.933926 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.934322 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.934323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.934323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.934428 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:56.934505 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:56.934663 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 15:22:56.934732 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 15:22:56.934821 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.927393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.125:34071 (local address 127.0.0.1:54778)
0924 15:22:56.927644 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.927653 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:56.927718 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:56.927880 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:56.927881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:56.928016 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:56.928069 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:56.928070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.928758 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.928759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:56.929412 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:56.929414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.929952 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.929953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:56.930022 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.930394 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:56.930397 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:56.931693 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:56.933553 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:56.933554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:56.933555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:56.933766 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:56.934000 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:56.934001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:56.934002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:56.934059 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:56.934511 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:56.934512 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:56.934552 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 15:22:56.934637 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 15:22:56.934729 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
I20260924 15:22:56.935117 22534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } has no permanent_uuid. Determining permanent_uuid...
I20260924 15:22:56.936123 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.926597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:35095 (local address 127.0.0.1:59222)
0924 15:22:56.926634 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.926644 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:56.926678 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:56.927076 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:56.927078 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:56.927260 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:56.927302 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:56.927304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.928185 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.928186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:56.928957 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:56.928958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.929397 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.929398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:56.929473 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.929836 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:56.929840 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:56.931278 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:56.933139 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:56.933140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:56.933141 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:56.933396 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:56.934119 (+ 723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:56.934120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:56.934120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:56.934176 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:56.935861 (+ 1685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:56.935862 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:56.935905 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 15:22:56.935949 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 15:22:56.936032 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 15:22:56.936513 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.927612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42172 (local address 127.31.12.126:38605)
0924 15:22:56.927646 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:56.927648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:56.927655 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:22:56.927697 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:56.927698 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:56.927706 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:56.927757 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:56.928256 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.928925 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.929754 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.930019 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.930083 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.933224 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:56.933226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:56.933226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:56.933233 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:56.934878 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.935457 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.935458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.935458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.935508 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.935698 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.935699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.935700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.935843 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:56.936022 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:56.936354 (+ 332us) server_negotiation.cc:299] Negotiation successful
0924 15:22:56.936427 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 15:22:56.936676 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.926862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59222 (local address 127.31.12.124:35095)
0924 15:22:56.926919 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:56.926921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:56.926928 (+ 7us) server_negotiation.cc:407] Connection header received
0924 15:22:56.926944 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:56.926945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:56.926967 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:56.927368 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:56.927396 (+ 28us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.928090 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.929026 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.929311 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.929374 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.931434 (+ 2060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:56.931435 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:56.931435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:56.931442 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:56.933036 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.933657 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.933658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.933658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.933745 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.934272 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.934272 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.934273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.935789 (+ 1516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:56.936400 (+ 611us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:56.936441 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 15:22:56.936511 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 15:22:56.938446 22382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320: Bootstrap starting.
I20260924 15:22:56.939153 22382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320: Neither blocks nor log segments found. Creating new log.
Sep 24 15:22:56 dist-test-slave-zwbb krb5kdc[22301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263376, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.124@KRBTEST.COM for kudu/127.31.12.124@KRBTEST.COM
I20260924 15:22:56.939481 22382 log.cc:824] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:56.940321 22382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320: No bootstrap required, opened a new log
I20260924 15:22:56.942265 22382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e972e31250440ea7b6c42bf8fd2320" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } } peers { permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } } peers { permanent_uuid: "e7e92d7a380048d39aacecc1e19a599e" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } }
I20260924 15:22:56.942412 22382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71e972e31250440ea7b6c42bf8fd2320, State: Initialized, Role: FOLLOWER
I20260924 15:22:56.942487 22382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e972e31250440ea7b6c42bf8fd2320" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } } peers { permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } } peers { permanent_uuid: "e7e92d7a380048d39aacecc1e19a599e" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } }
I20260924 15:22:56.942586 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.935366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:35095 (local address 127.0.0.1:59238)
0924 15:22:56.935468 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.935478 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:56.935509 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:56.935770 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:56.935771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:56.935915 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:56.935946 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:56.935948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.936915 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.936916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:56.937612 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:56.937615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.938108 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.938109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:56.938162 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.938560 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:56.938565 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:56.939791 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:56.941567 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:56.941568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:56.941569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:56.941779 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:56.941963 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:56.941963 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:56.941964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:56.942019 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:56.942312 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:56.942313 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:56.942345 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 15:22:56.942412 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 15:22:56.942496 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":99,"threads_started":1}
I20260924 15:22:56.942783 22382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:56.943070 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.935552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59238 (local address 127.31.12.124:35095)
0924 15:22:56.935654 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:56.935656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:56.935662 (+ 6us) server_negotiation.cc:407] Connection header received
0924 15:22:56.935679 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:56.935680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:56.935689 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:56.936005 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:56.936052 (+ 47us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.936829 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.937690 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.938029 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.938964 (+ 935us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.939881 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:56.939883 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:56.939883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:56.939889 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:56.941485 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.941853 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.941853 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.941854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.941901 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.942081 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.942082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.942082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.942234 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:56.942292 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:56.942877 (+ 585us) server_negotiation.cc:299] Negotiation successful
0924 15:22:56.942950 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":61,"threads_started":1}
W20260924 15:22:56.944490 22555 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:22:56.944538 22555 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:22:56.944515 22534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e: Bootstrap starting.
I20260924 15:22:56.945267 22534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:56.945626 22534 log.cc:824] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:56.946419 22534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e: No bootstrap required, opened a new log
Sep 24 15:22:56 dist-test-slave-zwbb krb5kdc[22301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.31.12.125@KRBTEST.COM
I20260924 15:22:56.948704 22534 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e972e31250440ea7b6c42bf8fd2320" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } } peers { permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } } peers { permanent_uuid: "e7e92d7a380048d39aacecc1e19a599e" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } }
I20260924 15:22:56.948838 22534 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e92d7a380048d39aacecc1e19a599e, State: Initialized, Role: FOLLOWER
I20260924 15:22:56.948908 22534 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e972e31250440ea7b6c42bf8fd2320" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } } peers { permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } } peers { permanent_uuid: "e7e92d7a380048d39aacecc1e19a599e" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } }
I20260924 15:22:56.949185 22534 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:56.947126 22314 master_runner.cc:186] Error getting master registration for 127.31.12.125:34071: OK, instance_id { permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" instance_seqno: 1790263376740573 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 15:22:56.949928 22314 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 15:22:56.950654 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.939744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54784 (local address 127.31.12.125:34071)
0924 15:22:56.939783 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:56.939785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:56.939793 (+ 8us) server_negotiation.cc:407] Connection header received
0924 15:22:56.939814 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:56.939815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:56.939823 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:56.939875 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:56.942079 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.942765 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.944107 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.944377 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.944422 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.947514 (+ 3092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:56.947515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:56.947516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:56.947522 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:56.949220 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.949665 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.949666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.949666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.949729 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.949977 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.949978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.949978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.950105 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:56.950247 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:56.950478 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 15:22:56.950564 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 15:22:56.951030 22558 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e972e31250440ea7b6c42bf8fd2320" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } } peers { permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } } peers { permanent_uuid: "e7e92d7a380048d39aacecc1e19a599e" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } } }
I20260924 15:22:56.951128 22558 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:22:56.951478 22542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e972e31250440ea7b6c42bf8fd2320" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } } peers { permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } } peers { permanent_uuid: "e7e92d7a380048d39aacecc1e19a599e" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } } }
I20260924 15:22:56.951572 22542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:22:56.952132 31793 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 15:22:56.952519 22569 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 15:22:56.952558 22569 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:22:56.953109 22356 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "71e972e31250440ea7b6c42bf8fd2320"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42172
I20260924 15:22:56.953238 22356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 15:22:56.953277 22356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:56.954169 22356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e972e31250440ea7b6c42bf8fd2320" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } } peers { permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } } peers { permanent_uuid: "e7e92d7a380048d39aacecc1e19a599e" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } }
I20260924 15:22:56.954412 22356 leader_election.cc:288] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [CANDIDATE]: Term 1 election: Requested vote from peers 0cd8c269cf3643669d5ca88a33f0bfe2 (127.31.12.125:34071), e7e92d7a380048d39aacecc1e19a599e (127.31.12.124:35095)
I20260924 15:22:56.954741 22508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71e972e31250440ea7b6c42bf8fd2320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e7e92d7a380048d39aacecc1e19a599e"
I20260924 15:22:56.954833 22508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:56.955695 22508 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e972e31250440ea7b6c42bf8fd2320 in term 1.
I20260924 15:22:56.955957 22331 leader_election.cc:302] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71e972e31250440ea7b6c42bf8fd2320, e7e92d7a380048d39aacecc1e19a599e; no voters:
I20260924 15:22:56.956070 22542 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 15:22:56.956214 22542 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [term 1 LEADER]: Becoming Leader. State: Replica: 71e972e31250440ea7b6c42bf8fd2320, State: Running, Role: LEADER
I20260924 15:22:56.956292 22542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e972e31250440ea7b6c42bf8fd2320" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } } peers { permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } } peers { permanent_uuid: "e7e92d7a380048d39aacecc1e19a599e" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } }
I20260924 15:22:56.956792 22575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71e972e31250440ea7b6c42bf8fd2320. Latest consensus state: current_term: 1 leader_uuid: "71e972e31250440ea7b6c42bf8fd2320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e972e31250440ea7b6c42bf8fd2320" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } } peers { permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } } peers { permanent_uuid: "e7e92d7a380048d39aacecc1e19a599e" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } } }
I20260924 15:22:56.956933 22575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:56.957679 22466 master_runner.cc:186] Error getting master registration for 127.31.12.125:34071: OK, instance_id { permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" instance_seqno: 1790263376740573 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 15:22:56.960002 22466 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 15:22:56.962021 22577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 15:22:56.962200 22577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 15:22:56.963819 22508 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e [term 1 FOLLOWER]: Refusing update from remote peer 71e972e31250440ea7b6c42bf8fd2320: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:22:56.964125 22542 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7e92d7a380048d39aacecc1e19a599e" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:22:56.965325 22558 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71e972e31250440ea7b6c42bf8fd2320. Latest consensus state: current_term: 1 leader_uuid: "71e972e31250440ea7b6c42bf8fd2320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e972e31250440ea7b6c42bf8fd2320" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } } peers { permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } } peers { permanent_uuid: "e7e92d7a380048d39aacecc1e19a599e" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } } }
I20260924 15:22:56.965425 22558 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:22:56.973433 22463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.961993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54800 (local address 127.31.12.125:34071)
0924 15:22:56.962030 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:56.962032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:56.963055 (+ 1023us) server_negotiation.cc:407] Connection header received
0924 15:22:56.963197 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:56.963198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:56.963208 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:56.963236 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:56.965537 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.966272 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.967297 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.967616 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.967681 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.970584 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:56.970585 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:56.970586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:56.970592 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:56.972209 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.972574 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.972575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.972575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.972624 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.972834 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.972835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.972836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.972967 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:56.973037 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:56.973251 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 15:22:56.973344 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 15:22:56.974028 22459 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } attempt: 3
I20260924 15:22:56.975975 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.961772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42176 (local address 127.31.12.126:38605)
0924 15:22:56.961812 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:56.961814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:56.962973 (+ 1159us) server_negotiation.cc:407] Connection header received
0924 15:22:56.963295 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:56.963296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:56.963305 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:56.963331 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:56.967311 (+ 3980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.968020 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.969016 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.969302 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.969364 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.972058 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:56.972059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:56.972059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:56.972066 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:56.973878 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.974854 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.974855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.974856 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.974908 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.975294 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.975295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.975295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.975426 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:56.975492 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:56.975816 (+ 324us) server_negotiation.cc:299] Negotiation successful
0924 15:22:56.975891 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:22:56.978587 22575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71e972e31250440ea7b6c42bf8fd2320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e972e31250440ea7b6c42bf8fd2320" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } } peers { permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } } peers { permanent_uuid: "e7e92d7a380048d39aacecc1e19a599e" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } } }
I20260924 15:22:56.978667 22575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [sys.catalog]: This master's current role is: LEADER
Sep 24 15:22:56 dist-test-slave-zwbb krb5kdc[22301](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790263376, etypes {rep=17 tkt=17 ses=17}, kudu/127.31.12.125@KRBTEST.COM for kudu/127.31.12.124@KRBTEST.COM
I20260924 15:22:56.978835 22581 mvcc.cc:205] Tried to move back new op lower bound from 7332918792041304064 to 7332918792013611008. Current Snapshot: MvccSnapshot[applied={T|T < 7332918792041304064}]
I20260924 15:22:56.979169 22558 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71e972e31250440ea7b6c42bf8fd2320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e972e31250440ea7b6c42bf8fd2320" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } } peers { permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } } peers { permanent_uuid: "e7e92d7a380048d39aacecc1e19a599e" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } } }
I20260924 15:22:56.979254 22558 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:22:56.982615 22461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.974237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.31.12.124:35095 (local address 127.0.0.1:59248)
0924 15:22:56.974343 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 15:22:56.974355 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 15:22:56.974394 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 15:22:56.974648 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 15:22:56.974649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 15:22:56.974794 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 15:22:56.974832 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:56.974833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.975746 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.975747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:56.976528 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 15:22:56.976530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.977242 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.977244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 15:22:56.977299 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.977680 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 15:22:56.977685 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 15:22:56.979664 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 15:22:56.981524 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:56.981525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:56.981525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:56.981749 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:56.981949 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 15:22:56.981950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 15:22:56.981950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 15:22:56.982008 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 15:22:56.982292 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 15:22:56.982293 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 15:22:56.982335 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 15:22:56.982397 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 15:22:56.982492 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260924 15:22:56.982987 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.974490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59248 (local address 127.31.12.124:35095)
0924 15:22:56.974524 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:56.974525 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:56.974531 (+ 6us) server_negotiation.cc:407] Connection header received
0924 15:22:56.974547 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:56.974548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:56.974555 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:56.974614 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:56.974911 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.975658 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.976744 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.977062 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.977140 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.979756 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:56.979757 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:56.979757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:56.979763 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:56.981415 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.981826 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.981827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.981827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.981879 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.982089 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.982090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.982090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.982211 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:56.982267 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:56.982796 (+ 529us) server_negotiation.cc:299] Negotiation successful
0924 15:22:56.982871 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:22:56.984323 22459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2: Bootstrap starting.
I20260924 15:22:56.984688 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:56.962687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59246 (local address 127.31.12.124:35095)
0924 15:22:56.962725 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:56.962727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:56.969329 (+ 6602us) server_negotiation.cc:407] Connection header received
0924 15:22:56.970883 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:56.970884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:56.970897 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:56.970927 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:56.978444 (+ 7517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.979163 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.980204 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:56.980513 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:56.980551 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:56.981978 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 15:22:56.981980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 15:22:56.981980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 15:22:56.981988 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 15:22:56.983606 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.983973 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.983974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.983974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.984032 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 15:22:56.984214 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 15:22:56.984215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 15:22:56.984215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 15:22:56.984314 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 15:22:56.984380 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:56.984543 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 15:22:56.984611 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:22:56.985009 22459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2: Neither blocks nor log segments found. Creating new log.
I20260924 15:22:56.985283 22459 log.cc:824] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2: Log is configured to *not* fsync() on all Append() calls
I20260924 15:22:56.985499 22577 catalog_manager.cc:1347] Generated new cluster ID: 8fb0acd3dbaf4392b086d443dd0ce7c8
I20260924 15:22:56.985543 22577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 15:22:56.994083 22459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2: No bootstrap required, opened a new log
I20260924 15:22:56.996177 22459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e972e31250440ea7b6c42bf8fd2320" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } } peers { permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } } peers { permanent_uuid: "e7e92d7a380048d39aacecc1e19a599e" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } }
I20260924 15:22:56.996313 22459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cd8c269cf3643669d5ca88a33f0bfe2, State: Initialized, Role: FOLLOWER
I20260924 15:22:56.996394 22459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e972e31250440ea7b6c42bf8fd2320" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } } peers { permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } } peers { permanent_uuid: "e7e92d7a380048d39aacecc1e19a599e" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } }
I20260924 15:22:56.996730 22459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 15:22:56.996774 22433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "71e972e31250440ea7b6c42bf8fd2320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2"
I20260924 15:22:56.996860 22433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 15:22:56.997738 22433 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71e972e31250440ea7b6c42bf8fd2320 in term 1.
I20260924 15:22:56.997947 22432 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2 [term 1 FOLLOWER]: Refusing update from remote peer 71e972e31250440ea7b6c42bf8fd2320: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 15:22:56.998373 22542 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 15:22:56.996701 22587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e972e31250440ea7b6c42bf8fd2320" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } } peers { permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } } peers { permanent_uuid: "e7e92d7a380048d39aacecc1e19a599e" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } } }
I20260924 15:22:57.003064 22587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:22:57.003160 22587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71e972e31250440ea7b6c42bf8fd2320. Latest consensus state: current_term: 1 leader_uuid: "71e972e31250440ea7b6c42bf8fd2320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e972e31250440ea7b6c42bf8fd2320" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } } peers { permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } } peers { permanent_uuid: "e7e92d7a380048d39aacecc1e19a599e" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } } }
I20260924 15:22:57.003254 22587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:22:57.003319 22587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "71e972e31250440ea7b6c42bf8fd2320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e972e31250440ea7b6c42bf8fd2320" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } } peers { permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } } peers { permanent_uuid: "e7e92d7a380048d39aacecc1e19a599e" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } } }
I20260924 15:22:57.003412 22587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 15:22:57.004524 22600 mvcc.cc:205] Tried to move back new op lower bound from 7332918792041304064 to 7332918792013611008. Current Snapshot: MvccSnapshot[applied={T|T < 7332918792041304064}]
I20260924 15:22:57.005766 22598 catalog_manager.cc:1259] Loaded cluster ID: 8fb0acd3dbaf4392b086d443dd0ce7c8
I20260924 15:22:57.005852 22598 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2: loading cluster ID for follower catalog manager: success
W20260924 15:22:57.005990 22598 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260924 15:22:57.006070 22598 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260924 15:22:57.006397 22542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "71e972e31250440ea7b6c42bf8fd2320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71e972e31250440ea7b6c42bf8fd2320" member_type: VOTER last_known_addr { host: "127.31.12.126" port: 38605 } } peers { permanent_uuid: "0cd8c269cf3643669d5ca88a33f0bfe2" member_type: VOTER last_known_addr { host: "127.31.12.125" port: 34071 } } peers { permanent_uuid: "e7e92d7a380048d39aacecc1e19a599e" member_type: VOTER last_known_addr { host: "127.31.12.124" port: 35095 } } }
I20260924 15:22:57.006495 22542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320 [sys.catalog]: This master's current role is: LEADER
I20260924 15:22:57.007911 22577 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 15:22:57.008611 22577 catalog_manager.cc:1504] Loading token signing keys...
I20260924 15:22:57.015035 22577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71e972e31250440ea7b6c42bf8fd2320: Generated new TSK 0
I20260924 15:22:57.015190 22577 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 15:22:57.038030 22463 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:57.036415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54814 (local address 127.31.12.125:34071)
0924 15:22:57.036462 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:57.036464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:57.037339 (+ 875us) server_negotiation.cc:407] Connection header received
0924 15:22:57.037493 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:57.037494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:57.037506 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:57.037537 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:57.037965 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54814: BlockingRecv error: recv got EOF from 127.0.0.1:54814 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 15:22:57.038939 22385 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:57.036802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42192 (local address 127.31.12.126:38605)
0924 15:22:57.036840 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:57.036842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:57.038443 (+ 1601us) server_negotiation.cc:407] Connection header received
0924 15:22:57.038539 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:57.038540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:57.038550 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:57.038579 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:57.038877 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42192: BlockingRecv error: recv got EOF from 127.0.0.1:42192 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 15:22:57.039847 22540 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:57.037082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59264 (local address 127.31.12.124:35095)
0924 15:22:57.037120 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:57.037122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:57.039322 (+ 2200us) server_negotiation.cc:407] Connection header received
0924 15:22:57.039461 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:57.039462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:57.039473 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:57.039502 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:57.039788 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59264: BlockingRecv error: recv got EOF from 127.0.0.1:59264 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 15:22:57.042287 22463 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:57.041576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.31.12.125:34071)
0924 15:22:57.041610 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:57.041611 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:57.041806 (+ 195us) server_negotiation.cc:407] Connection header received
0924 15:22:57.041908 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:57.041909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:57.041925 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:57.041954 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:57.042242 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54818: BlockingRecv error: recv got EOF from 127.0.0.1:54818 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 15:22:57.045313 22463 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:57.044585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54832 (local address 127.31.12.125:34071)
0924 15:22:57.044615 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:57.044617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:57.044838 (+ 221us) server_negotiation.cc:407] Connection header received
0924 15:22:57.044956 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:57.044957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:57.044966 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:57.044993 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:57.045270 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54832: BlockingRecv error: recv got EOF from 127.0.0.1:54832 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 15:22:57.049017 22463 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:57.048373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54836 (local address 127.31.12.125:34071)
0924 15:22:57.048412 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:57.048414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:57.048607 (+ 193us) server_negotiation.cc:407] Connection header received
0924 15:22:57.048689 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:57.048690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:57.048699 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:57.048723 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:57.048980 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54836: BlockingRecv error: recv got EOF from 127.0.0.1:54836 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 15:22:57.054819 22463 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:57.054108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54838 (local address 127.31.12.125:34071)
0924 15:22:57.054138 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:57.054140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:57.054342 (+ 202us) server_negotiation.cc:407] Connection header received
0924 15:22:57.054442 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:57.054443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:57.054452 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:57.054478 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:57.054779 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54838: BlockingRecv error: recv got EOF from 127.0.0.1:54838 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 15:22:57.064642 22463 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:57.063869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54840 (local address 127.31.12.125:34071)
0924 15:22:57.063904 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:57.063906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:57.064132 (+ 226us) server_negotiation.cc:407] Connection header received
0924 15:22:57.064224 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:57.064224 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:57.064234 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:57.064261 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:57.064585 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54840: BlockingRecv error: recv got EOF from 127.0.0.1:54840 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 15:22:57.082998 22463 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:57.082069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54856 (local address 127.31.12.125:34071)
0924 15:22:57.082111 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:57.082113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:57.082416 (+ 303us) server_negotiation.cc:407] Connection header received
0924 15:22:57.082525 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:57.082527 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:57.082538 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:57.082568 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:57.082941 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54856: BlockingRecv error: recv got EOF from 127.0.0.1:54856 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 15:22:57.117354 22463 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:57.116457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54872 (local address 127.31.12.125:34071)
0924 15:22:57.116502 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:57.116504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:57.116794 (+ 290us) server_negotiation.cc:407] Connection header received
0924 15:22:57.116897 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:57.116898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:57.116909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:57.116938 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:57.117296 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54872: BlockingRecv error: recv got EOF from 127.0.0.1:54872 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 15:22:57.183724 22463 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:57.182942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54874 (local address 127.31.12.125:34071)
0924 15:22:57.183024 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:57.183026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:57.183164 (+ 138us) server_negotiation.cc:407] Connection header received
0924 15:22:57.183220 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:57.183221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:57.183231 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:57.183283 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:57.183644 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54874: BlockingRecv error: recv got EOF from 127.0.0.1:54874 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 15:22:57.313773 22463 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:57.312959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54888 (local address 127.31.12.125:34071)
0924 15:22:57.313031 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:57.313034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:57.313119 (+ 85us) server_negotiation.cc:407] Connection header received
0924 15:22:57.313224 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:57.313225 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:57.313235 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:57.313288 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:57.313693 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54888: BlockingRecv error: recv got EOF from 127.0.0.1:54888 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260924 15:22:57.571844 22463 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 15:22:57.571036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54896 (local address 127.31.12.125:34071)
0924 15:22:57.571097 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:57.571100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:57.571135 (+ 35us) server_negotiation.cc:407] Connection header received
0924 15:22:57.571238 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:57.571239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:57.571250 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:57.571342 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 15:22:57.571786 (+ 444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54896: BlockingRecv error: recv got EOF from 127.0.0.1:54896 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 15:22:57.953006 22569 catalog_manager.cc:1259] Loaded cluster ID: 8fb0acd3dbaf4392b086d443dd0ce7c8
I20260924 15:22:57.953075 22569 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e: loading cluster ID for follower catalog manager: success
I20260924 15:22:57.953976 22569 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e: acquiring CA information for follower catalog manager: success
I20260924 15:22:57.954273 22569 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e7e92d7a380048d39aacecc1e19a599e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:22:58.007194 22598 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2: acquiring CA information for follower catalog manager: success
I20260924 15:22:58.007572 22598 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0cd8c269cf3643669d5ca88a33f0bfe2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 15:22:58.087769 22671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:58.084970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54902 (local address 127.31.12.125:34071)
0924 15:22:58.085102 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:58.085105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:58.085183 (+ 78us) server_negotiation.cc:407] Connection header received
0924 15:22:58.085285 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:58.085286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:58.085357 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:58.085443 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:22:58.085642 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:58.086152 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:58.086945 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:58.087167 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:58.087243 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:58.087417 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:22:58.087532 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:22:58.087619 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:58.087643 (+ 24us) server_negotiation.cc:299] Negotiation successful
0924 15:22:58.087666 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260924 15:22:58.091852 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:58.089682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59270 (local address 127.31.12.124:35095)
0924 15:22:58.089806 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:58.089808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:58.089847 (+ 39us) server_negotiation.cc:407] Connection header received
0924 15:22:58.089924 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:58.089925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:58.089961 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:58.090027 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:22:58.090099 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:58.090549 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:58.091183 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:58.091364 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:58.091420 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:58.091540 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:22:58.091625 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:22:58.091677 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:58.091715 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 15:22:58.091730 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260924 15:22:58.096278 22671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:58.093423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54914 (local address 127.31.12.125:34071)
0924 15:22:58.093492 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:58.093493 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:58.093709 (+ 216us) server_negotiation.cc:407] Connection header received
0924 15:22:58.093784 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:58.093785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:58.093793 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:58.093842 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:22:58.093920 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:58.094547 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:58.095557 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:58.095813 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:58.095861 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:58.096044 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:22:58.096074 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:22:58.096120 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:58.096176 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 15:22:58.096194 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 15:22:58.096576 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:58.093466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42194 (local address 127.31.12.126:38605)
0924 15:22:58.093695 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:58.093697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:58.093708 (+ 11us) server_negotiation.cc:407] Connection header received
0924 15:22:58.093771 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:58.093772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:58.093815 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:58.093895 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:22:58.094010 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:58.094716 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:58.095439 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:58.095712 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:58.095774 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:58.095979 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:22:58.096079 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:22:58.096135 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:58.096470 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 15:22:58.096493 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":55,"threads_started":1}
I20260924 15:22:58.096639 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 15:22:58.093625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59286 (local address 127.31.12.124:35095)
0924 15:22:58.093663 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 15:22:58.093664 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 15:22:58.094013 (+ 349us) server_negotiation.cc:407] Connection header received
0924 15:22:58.094079 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 15:22:58.094081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 15:22:58.094089 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 15:22:58.094126 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 15:22:58.094232 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:58.094717 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:58.095685 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 15:22:58.095939 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 15:22:58.096010 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 15:22:58.096187 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 15:22:58.096218 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 15:22:58.096245 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0924 15:22:58.096510 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 15:22:58.096540 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 15:22:58.098269 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 22314
I20260924 15:22:58.103998 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 22389
I20260924 15:22:58.109148 31793 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskqOs6Q8/build/release/bin/kudu with pid 22466
2026-09-24T15:22:58Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2787 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5549 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (297810 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
2 FAILED TESTS